Cleaning source code folders for uploading

coldandtired

After many hours comparing and deleting folders full of source code (even worse than x, x.old, x.really.old) I've set up a Bitbucket account with private repos and am now nearly ready to upload some projects.

However, I've tended to jump around between IDEs and each project folder can contain files from IntelliJ, Eclipse, Netbeans, VS Code, etc. and I have code in multiple languages.

So, what are the best practices for uploading? At the moment it's mainly as a backup solution for myself but if I ever shared any of the projects I would like them not to leak personal (or useless) information.

I've found a website that can generate a .gitignore file containing rules for every IDE/OS/language - should I just make one universal file for all my needs and use it everywhere?

gitignore.io

Create useful .gitignore files for your project

pie_flavor

@coldandtired Yeah, or just type it yourself.

.idea/
.vs/
.gradle/
build/
out/
target/
run/
bin/
_MACOSX/
*.iml
.classpath
.project
.DS_Store
Thumbs.db

is generally the list I use in my Gradle Java projects. I wouldn't recommend doing a big global one - only use things you expect to see.

Choonster

@coldandtired You can also create a whitelist-style .gitignore like this that ignores everything (the first line) except the files you actually want in the repository (the lines starting with an exclamation mark).

Zecc

Or just add the files/directories you want explicitly (d). Though that might potentially be more work.

Just be careful not to upload private keys, login details, connection strings, etc. that you might have distractedly hard-coded.

dkf

@coldandtired said in Cleaning source code folders for uploading:

So, what are the best practices for uploading?

I recommend never uploading IDE configurations as such, but rather just those files that other people could reasonably use without modification. Also, almost never upload binaries and libraries (there are a few rare exceptions to this, but you probably aren't dealing with them) and never upload outputs. Particularly not 10GB CSV files like what some of our students tried to do (and failed at because github has at least some limits, thank God!) and which forced me to fix their repository so they could share their work once more…

BEWARE of marking every file for uploading. Some tools are set up to work that way by default, but it is a way of getting into problems; it's better to take a moment to check that your change is what you think it is before committing instead of having to patch up your stupidity after the fact… Speaking as the stupid one on too many occasions! ;)

Onyx

@dkf said in Cleaning source code folders for uploading:

I recommend never uploading IDE configurations as such, but rather just those files that other people could reasonably use without modification.

I guess stuff like .eslint files that include the style rules and such are in that latter group, for example. Or were you thinking about something else?

blakeyrat

@Onyx In Visual Studio terms, it means include the .sln and .proj files, but don't include the .user files. The former two carry definitions needed for all users, but the latter contains stuff relevant only for yourself.

I dunno how other IDEs do it. I've heard that Eclipse mixes stuff together in a giant mish-mash.

Onyx

@blakeyrat said in Cleaning source code folders for uploading:

@Onyx In Visual Studio terms, it means include the .sln and .proj files, but don't include the .user files.

I dunno how other IDEs do it.

We use VSCode but .vsproj (or whatever they are called) don't really make sense given how we develop so even those are out. I don't upload the .user file for the C++ portion of the stack, but I do upload the .pro files because they are used by the build system as well, even if the IDE isn't involved at all (I assume that's how building works with .sln files as well?).

The reason I asked about stuff like .eslint which is IDE-agnostic is it's kind of an edge case, so I'm wondering what people do with those. We upload them, and if the IDE/editor you're using is configured to use eslint it will use it, otherwise it's just sitting there and bothering no one, imho.

ben_lubar

@blakeyrat Visual Studio makes it pretty easy by automatically creating a .gitignore file for you.

HOWEVER, any git repository that contains text files with CRLF line endings or non-extended-ASCII text files internally is hopelessly broken.

Want to store PowerShell scripts in little-endian UTF-16 with no byte order mark and CRLF line endings? Add this to .gitattributes:

*.ps1		text working-tree-encoding=UTF-16LE eol=CRLF

That makes Git convert the files to UTF-8 with LF line endings when committing and convert back when checking out.

The reason I say "hopelessly broken" is that any time you get a "this file has mixed line endings" warning in Visual Studio when opening a file, you probably have some people committing with CRLF line endings and some people with core.autocrlf=input set, which means to commit LF line endings but leave checked-out files as-is.

Once you have text files that are not UTF-8 with LF line endings in git history, you basically have three choices:

Fix .gitattributes and then renormalize the entire repository, which changes the "last modified" date on every single file and will cause tons of merge conflicts if there are other open branches
Live with bullshit like I described above forever.
Convert the history to have a proper .gitattributes in it from the start, which will force everyone to throw away their local copy because it is a lot easier to just clone it from scratch than it would be to fix it.

ben_lubar

@ben_lubar actually, come to think of it, there is a fourth option: add a .gitattributes file that does what you want and then just let files get fixed naturally over time as they are modified.

This will make the first diff for each file horrible, but at least each file will be modified with an actual modification.

dcon

@ben_lubar said in Cleaning source code folders for uploading:

you probably have some people committing with CRLF line endings and some people with core.autocrlf=input set, which means to commit LF line endings but leave checked-out files as-is.

goddamnfuckinggit.

EOLs should be an attribute of a file - not a fucking client.
Oh wait, the git-rant thread is

@ben_lubar said in Cleaning source code folders for uploading:

Live with bullshit like I described above forever.

(well, until I find a new job...)

cartman82

On shared projects, I like to add some IDE configuration files to source control. The point is to add coding style stuff and run configurations, and leave out user-specific settings.

For JetBrains editors, I use this: https://github.com/github/gitignore/blob/master/Global/JetBrains.gitignore
Works reasonably well.

I don't have anything handy for VSCode, I usually use it as a text editor and don't cross-load projects with it.

However, in your situation, it sounds like your repos are 100% private and you don't intend to share them with anyone else. In that case, I'd go right ahead and add all editor files to directly VC. Practicality beats theoretical purity.

dkf

@Onyx said in Cleaning source code folders for uploading:

I guess stuff like .eslint files that include the style rules and such are in that latter group, for example.

I guess so? My projects use (for preference) a different structure that makes that not directly applicable, but I think it's the sort of thing I meant.

dkf

@blakeyrat said in Cleaning source code folders for uploading:

I've heard that Eclipse mixes stuff together in a giant mish-mash.

Never that I've noticed, at least not with conventional projects where you don't lock them into being Eclipse-specific. (I don't want my projects to require any IDE about to build; that stuff interferes with CI servers and so on.)

dkf

@cartman82 said in Cleaning source code folders for uploading:

However, in your situation, it sounds like your repos are 100% private and you don't intend to share them with anyone else. In that case, I'd go right ahead and add all editor files to directly VC. Practicality beats theoretical purity.

But don't add any files with credentials in if there is the slightest chance that the repo may ever become public. Or even just get shared with one other person.

cartman82

@dkf said in Cleaning source code folders for uploading:

But don't add any files with credentials in if there is the slightest chance that the repo may ever become public. Or even just get shared with one other person.

But aren't those gonna be just garbage localhost passwords and crap like that? How many production credentials are a part of your editor files really?

PleegWat

@ben_lubar

On that front, I've been considering doing whitespace normalization (get rid of tabs in basically all files except Makefiles) but I'm likewise afraid of the diff consequences. Not git either though.

heterodox

@coldandtired said in Cleaning source code folders for uploading:

if I ever shared any of the projects I would like them not to leak personal (or useless) information.

I'm surprised it hasn't been mentioned yet but if you're worried about personal information (this thread has drifted a bit), you should obviously check the name and email address on your commits and --reset-author as needed.

boomzilla

@blakeyrat said in Cleaning source code folders for uploading:

I've heard that Eclipse mixes stuff together in a giant mish-mash.

Yes, that's possible, although you can use variables in all your classpath stuff which people then set based on their local environments. Like so many things, it requires a little bit of thought so that you're not .

Tsaukpaetra

@cartman82 said in Cleaning source code folders for uploading:

How many production credentials are a part of your editor files really?

*Pleads the fifth*

The current way we're baking connection strings into the game necessarily lives within configurations that aren't separable into alternatives files.

I'm working on it though, but the automation still requires them at some point...