I. Project management and documentation#
The goal of this first module is to get familiar with the use of softwares using CLI (from Command Line Interface), as well as the documentation and management of projects. I will describe here what I have learned how I did it.
1. CLI : Command Line Interface#
If most of the softwares we daily use are displayed with a user-friendly interface, called Graphical User Interface or GUI which allows for an easy handling for beginners, there exist many softwares using a Command Line Interface or CLI.
It is generally harder to acquire the basics on those because to use them you need to know a certain amount of commands, there is nothing you can just click on to carry out an action. However, they allow for lighter and faster softwares, and for the regulars, their handling can become quite fast.
2. Documentation and version control#
Documenting your learning processes is an excellent way to later be able to quickly remember or relearn what you did in the past. Should your documentation be online and accessible it then allows you to share your knowledge and tips you have learned with others.
For this purpose, the PHYS-F517 course makes use of a GitLab server where students can upload their files containing their documentation. The website is then generated by MkDocs, using those files.
2.1 Git#
GitLab provides you with an online editor, the Web IDE, that can be used to write your documentation directly on GitLab. However, should you prefer a more time-efficient way, it is recommended to create a clone (a sort of local copy) of the repository (a sort of centralised database accessible to different users) of your project. This clone is therefore stored offline on your computer and is faster to modify and use than this Web IDE.
Git is a software managing and controlling the versions of files and folders between a repository on a server (GitLab in this instance) and its clones. Git compares and tracks the different versions of a project. Hence when using Git correctly, users can create multiple clones (on several computers) of a same repository and yet still always work on the latest version of the project, even if that version was not made on their current computer.
2.1.1 Terminal#
Most of Git’s distributions do not have a graphical interface. To use it you have to call the software through commands on a terminal.
We must therefore choose an appropriate terminal. You could use the Windows Command Prompt, or PowerShell. I however settled for a the Linux built-in command prompt because most of the documentation on those kinds of subjects on the web refers to Linux, and the commands seemed simpler in the Linux terminal. My computer is nonetheless running on Windows and I did not intend to change that. Luckily, a feature exists in that purpose : the “Windows Subsystem for Linux”, or “WSL”.
You can install that feature quite easily by running a PowerShell terminal in admin mode and using the command : wsl --install.
This action actually installs both a WSL command prompt and a Ubuntu distribution (running only through a CLI), which both can be used for our purpose. At the first launch you will be asked to setup a username and a password. Firing up either of those softwares should provide you with a working Linux terminal.
2.1.2 Installation and setup#
Git being a free and open-source software you can find several distributions of it, like this one, where you can choose the appropriate version following your OS, or that one, specific to Windows computers. You can check whether Git is correctly installed by entering this command in your preferred terminal : git --version
Once Git is indeed installed, you then have to set it up. This configuration is needed for the program to recognise you as the author of your work. This is done by setting up a username and an email address like this :
git config --global user.name "your_username"
git config --global user.email "your_email_address@example.com"
However because I decided to use a WSL, to ensure that the configuration will be available from the WSL command prompt I must use it (the WSL terminal) to setup Git. Indeed, WSL installs itself on a kind of compartmentalised space on your drive, and has a limited access to the rest of it. This allows WSL to use Git even if it is installed on the usual part of the drive, but does not grant it access to the Git configuration file if it is not stored on that specific part of the drive.
You can then again check your configuration with the command : git config --global --list.
2.1.3 SSH Keys#
SSH Keys are a security protocol that can be used to securely communicate between a server and a client. This allows for this communication to be done without always having to log in again. In this instance, the server is GitLab and the client is the computer on which the clone you are working on is. SSH keys are created in pair : a private one that should be kept safe, and a public one.
2.1.3.1 Generating the keys#
As I was not yet used to CLI softwares, I preferred using the GUI Git I had installed to create my SSH keys.
In this GUI Git, the “Help” menu leads to a “Show SSH keys” tab that opens another window. As I did not yet have any SSH keys on my computer, the “Generate Key” button was available. Clicking on it automatically generates a pair of keys (id_rsa and id_rsa.pub), stored in a “.ssh” folder in your User’s main folder. The public key is also displayed in that window after its generation.
By using the GUI, I however could not choose where the keys were created and afterwards had to move the “.ssh” folder inside the WSL specific folder in order to make sure they were available for the Linux terminal.
2.1.3.2 Linking the keys to the server#
You then have to link your keys to your GitLab server. In order to do this, go on your user settings on your GitLab account, then proceed to the “SSH Key” tab. On the page, click on “Add a new key”.
The next step is to paste your public key in the designated box. You can find your public key in your “.ssh” repertory under the name “id_rsa.pub”. You should give a title or name to this key and verify that its usage type is indeed set on “Authentication & Signing” and that the expiration date suits you. Once it is all set, just confirm by clicking on “Add key” at the bottom of the page and you should be ready to go.
2.1.3.3 Key rights#
Unfortunately, probably due to the method I used to create my keys I encountered another issue. While I was trying to make my first clone (see section 2.1.4 Cloning) of the repository, I obtained the following error :
The permissions given to the other users (groups and public) were too open, which meant that my keys were unprotected. As you can see in the previous image, we get more information on this issue on the line “Permissions 0644 …”. Actually the numbers refer to certain permissions given to the different users. The second digit refers to the owner’s rights, a “6” meaning I can both read and write the file (being the private key here). The third digit refers to the permissions of the groups, while the fourth refers to the public’s permissions. Both of them have the “4” value, meaning they can read the key. This is obviously where the issue lies, as the private key is supposed to be private and not readable by everyone.
To resolve that issue I used the chmod command (see here for more information about its use) to change the permissions of the file. I therefore attributed the value “0” to the groups and public, which corresponds to no right at all on the file. This command requires to first place yourself in the directory where the file needing modifications is. Then type the command, its digit attributes for the rights and finally the name of the file you want to modify. It should be something like this :
cd /home/.shh
chmod 600 id_rsa
2.1.4 Cloning#
Once Git and your SSH keys are set up, you can use them to easily clone a repository on your computer. On the upper right corner of your project main page you will find a “Clone” button :
You can then copy the given URL to make a clone with SSH. In your command prompt then simply choose the folder where you want your clone to be and type the command :
git clone "URL"
where you should replace “URL” by the link you copied just earlier.
Git should then tell you the cloning has been completed.
2.1.5 Synchronising your versions#
After the cloning is completed, you can then modify your project locally. Once you have finished editing your project, or before and after substantial changes, you should save and synchronise your work. In order to do this you will have to follow steps, in the following order :
- In your command prompt, position yourself inside your project’s clone.
- Add your changes : with
git add X. You should replace “X” by the name of the folder or file you want to add, or alternatively type “-A” to add all your changes. - Commit your changes : with
git commit. You can add “-m” followed by a comment surrounded by quotations marks to annotate the changes you made, it can help you to track your different versions. - Finally, send your changes to the server with :
git push.
Should you be working with other people on a same project or should you have made modifications to your project on another computer or with the Web IDE, you then have to download the latest version of your project before making any local modifications. You can do this by using : git pull (after positioning yourself in the right directory).
You can always check on the status of your modifications and your add and commit (at whatever step you are) by typing : git status. The files displayed in red are the ones that went under local modifications and not yet been added. Once you have added your changes, those files should be displayed in green.
2.2 Modifying your project#
To modify your project via the clone, you just need to open the clone’s directory on your computer. There you can access all the files from your project.
2.2.1 Text editor#
Markdown is a sort of language used for many applications such as online forums, collaborative softwares and documentation pages. We will use this language to write our documentation in “.md” files. The language is quite simple, you nonetheless need a specific syntax to create lists, tables, etc. Here can you find the basic syntax, the extended syntax and a cheat sheet.
You should then ideally choose should a text editor that helps with this notation (eg : by having built-in Markdown preview or plugins). In this regard I chose to use Visual Studio Code that has a built-in Markdown preview panel and can also integrate the WSL Linux terminal thanks to its WSL extension feature.
Visual Studio Code allows you to open the whole folder containing your clone in order to visualise and access your folder’s structure and all of its files. From there you can modify your “.md” files and add images to your directories.
2.2.2 Images#
Implementing images to your documentation can be very helpful. It is also quite easy to do in Markdown. If your picture is included in your project the just type :
The picture description is optional and not visible in the rendered output. If your picture comes from the internet and you did not include it in your project’s folder you just have to replace the relative path with internet URL linking to the picture. However with this method, should the picture be (re)moved from the website it would no longer be displayed in your documentation.
You should however be conscious that uploading data on servers requires much energy and therefore try to reduce the size of the files you use. This can be done by reducing the quality of the pictures you upload with a software such as GIMP. Open your picture with software then in the “File” tab on the upper left corner of the window select “Export as”. If you export your picture as a “.jpg” file then you can choose the quality of your export, going from 0 to 100. A slight quality reduction can have a bigger impact on the file’s size. Exporting it as a “.png” has little interest because it is a lossless type of file and is unlikely to produce a lighter file.
You can also scale your images, this allows you to make sure they are displayed with a certain dimension on the website. This can be done with or without quality loss, depending on the used method :
- A lossless method constitutes of using HTML inside your Markdown (if it is supported). Instead of using the Markdown syntax use :
<img src="relative path" width="desired width in pixels" height="desired height in pixels">
- GIMP can also be used to resize (the dimension) your pictures : in the “Image” tab on the top of your window select the “Scale Image” menu. You can then choose the size of your picture (in pixels or in cm or in percents relative to the original size, …). You then have to export the resulting image.
- A really fast method (once set up), similar to the GIMP Image Scale, is the Image Resizer for Windows, now included in the Microsoft PowerToys available on the Microsoft Store. Once installed, they offer various features including an Image Resizer.
This feature is accessible by right-clicking on the picture you want to resize. It then presents itself a bit like the Scale Image menu of GIMP.
HTML can also be used to frame your pictures.
<img src="relative path" style="border:2px solid gray">
You can chose the thickness of your frame (in pixels) and its colour. You just have to change the number of pixels (px) and the name of the colour.
2.2.3 Videos#
When talking about the pictures, I mentioned you should avoid uploading too big files. While pictures can have sizes around several MB, videos can be much bigger. Hence it is particularly important to try and reduce their size before uploading them.
FFmpeg#
To edit videos I used FFmpeg. FFmpeg is a free and open-source software that provides a complete, cross-platform solution to record, convert, and stream audio and video. It is capable of converting video and audio in various formats, making it very useful. The website also provides comprehensive documentation to help users understand how to use the software. In addition to the official documentation available on this website you can also find different documentations from contributors. For example this useful cheat sheet was made by the Massachusetts Institute of Technology.
Here are some of its features :
- Support for Various Codecs and Filters : FFmpeg supports a wide range of codecs and filters, including Vulkan decoding, AV1 encoding, and many others.
- Command Line Interface : FFmpeg provides a command-line interface for users to interact with, making it flexible and powerful.
Actually, I did not install using the available downloads but rather directly using commands in my WSL. I used sudo apt update && sudo apt upgrade and sudo apt install ffmpeg, as it is explained here.
Muting a video : in your WSL console, type :
ffmpeg -i $input_file -c copy -an $output_file
where “$input_file” and “$output_file” are the name of your input and output files.
Reducing the size of a video : in my WSL I typed :
ffmpeg -i $input_file_ -vcodec libx265 -crf 28 $output_file
VLC Media Player#
Unfortunately, the video I recorded with my phone was in HEVC codec, which is not supported by Visual Studio Code and MkDocs (the site generator). As I was very unfamiliar with FFmpeg and its commands, I decided to use VCL Media Player to resolve this issue.
VLC Media Player is a free and open source cross-platform multimedia player and framework that plays most multimedia files as well as DVDs, Audio CDs, VCDs, and various streaming protocols. It supports various extensions but more importantly for us, it allows you to easily convert audio and video files format and codecs.
Once you opened the Convert window, add the file you want to edit and hit “Convert”. A new window will open :
You can then select a preexisting convert profile and edit it to your liking by pressing the wrench. Similarly you can create a new one from scratch and save it to reuse it later, by clicking on “Create a new profile” (next to the red cross). When creating a new profile or editing an existing one you can choose the encapsulation method (meaning the file extension : MP4/MKV/AVI/FLV/…), as well as the audio and video codecs. There you can choose among the list and specify the desired quality (bitrate, quality and frame rate). By unchecking the box on the top, you can disable the video or the audio (on the Audio codec tab).
Save your profile, choose the destination folder and the name of your modified video and press “Start” : your new video should be ready after a few moments (depending on its size).
Including a video in Markdown#
To include a video present in your repository in your Markdown file you have to use the html command :
<video width="desired_width" height="desired_height" controls src="relative_path_to_your_video">
video description
</video>
2.3 Building the site#
The website is build by MkDocs. MkDocs is a fast, simple and downright gorgeous static site generator that’s geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file. This will be automatically done for us 3 times a week, with the content of our GitLab repository. You can see them on GitLab under the Build tab : Pipeline schedules.
If you want to immediately see your modifications on the website, you can either press “Run pipeline schedule” or install MkDocs on your computer. As it is advised not to use too much the first method in order to stay frugal, I recommend you to install MkDocs. It is very handy to modify your images, videos, themes, as see your modifications in real time, and can immediately see what works and what does not.
If you wish to install it, you can use its Installation guide available on their website. Once it is installed, in a terminal, position yourself at the repository of your project and type mkdocs serve. It will generate an offline version of the website, accessible from your computer only (go to a web browser and paste the link available in the terminal), while this command is running. However this method will not take into account the instructions given in the “.gitlab-ci.yml” file. So, if you want to have another theme than the two defaults, you will have to manually install them (only once) via your terminal. For example :
sudo apt install mkdocs-bootswatch
will make accessible to you the different themes contained in the bootswatch extension.
3. Project management#
To help us get off to a good start, we learnt about some project management principles. You will find here below some of those that caught my attention.
3.1 As-your-work documentation#
As the course maintains a steady pace and we learn new things each week, it is important to regularly update our documentation. Indeed, writing it as it is still fresh in our memory is easier than trying to remember it some weeks later. It also helps us to realise whether we are late or not in our learning processes.
3.2 Modular and hierarchical planning#
Projects can sometimes appear as humongous hence scary tasks. To avoid procrastinating by fear of starting, it is a good practice to divide it into smaller, modular tasks you can easily take on.
3.3 Spiral development & triage#
You should keep an eye set on the available time to realise your project. Start by the essential (by using triage), and if you have the time, build around it, improve it. Should you run out of time, you would then have to select the tasks you must and/or can do in the remaining time.
3.4 Supply-side time management (SSTM)#
This is closely linked to the spiral development and the modular planning. SSTM is a time management method where instead of just going from one task to another hoping you will finish in time (demand-side time management), you break your total available time down into smaller periods (e.g. into weeks) and attribute a small task to each one of those. This helps you to make sure you keep up with the total allocated time for your project.
3.5 Time management laws#
Some empirical laws about time management have been observed and formulated over the years. Here are two of them :
- Parkinson’s Law : “Work expands so as to fill the time available for its completion.”
What I got out of it was that you should rather plan shorter than needed time periods to complete a task, because even if you plan a way too long period for the task, you will eventually drag it until the end of the planned period. - Hofstadter’s Law : “It always takes longer than expected, even taking into account Hofstadter’s Law.”
What I understood from it was that you should not plan a huge period of time for a task, hoping you will therefore necessarily have finished it then, because that probably will not be the case. To me it seemed a bit similar to the Parkinson’s Law, but I might not have understood it properly.