Linux kernel mentorship

Introduction:

    Welcome to my blog on Linux Kernel Mentorship experience. I was the part of kernel mentorship program summer 2025 with part time commitment. This mentorship program is designed by the Linux Foundation and happen in multiple terms throughout the year. More information about this program can be found in the link [1]. There are multiple blogs from graduated mentees from this program over the past few years [2]. I highly encourage you to have a glance on them as well. I have read many of them before joining this program. In this blog, I will mainly focus on my experience, thoughts and key learnings that can benefit many of those who want to start their upstream kernel contribution journey. I will not be going into details of any specific tools or technical things because many blogs already cover this and I am pretty sure that readers are curious to figure-out on their own using various online resources. Below is the 10000 feet view of things that we are going to cover in this blog. Keep reading on..

Why Linux kernel ? The Motivation:

    The linux kernel is the most successful and impactful open source project in the world. Linux kernel powers majority of the servers and supercomputers in the world. It is present in your smartphone, your smartwatches, your car, your laptop and many more devices around you. Most probably this blog is also hosted on the Linux powered server. So, if linux kernel is so important, what if you get a chance to contribute to it ? How will be the feeling if the code you wrote or debugged impacts millions of devices worldwide, including devices your personally use in day-to-day life ? Cool right ? You can proudly say to your parents, spouse, friends, colleagues or siblings that - hey, the phone that you are using runs my code (no matter how small it is 0.00000001% maybe). You impact the life of many people through your contribution, no matter how small it is, or big, but there will be an impact. That is exactly my motivation to contribute to the kernel ! And I hope many people may get motivation from this thought.

Myths about contributing to upstream kernel:

    When I started my career as an embedded software engineer, I mostly worked on bare metal C programming. I am active linux user since my high school. I remember experimenting with various kernel distributions like Ubuntu, Mint, Arch back then. I always though, kernel development is very complex task and one needs special training with a lot of OS knowledge to do so. This thought kept me away from touching the kernel development until I met with one of my colleges who is active upstream kernel contributor. His words "You need not to be expert to start kernel development, you need correct mindset and willingness to contribute".  I learned that even though kernel development is challenging, there is always a starting point to begin with. I got to know about this mentorship program and I decided to attempt - kernel contributions. I always had a question in my mind, okay I decided to contribute to the upstream kernel, but where do I start ? what I can contribute to ? Keep reading on ...

Few myths about upstream kernel development you should delete from your mind right now: 

- It is only for those who are expert computer scientist with lot of experience.

- Linux kernel is "done" thing. It is fully developed and there is nothing more to do.
- You need to be an expert of operating system concepts and theory. 
- You need to be an expert programmer.
- You can only find a place in the kernel community if you are working in reputed company or have good network with maintainers.
- Kernel is governed and influenced by few big and influential companies.

If you have any of the above thoughts, please remove those, these are only myths according to me.

Where to start:

    This is the common question from any newcomer, including me. This is probably the most obvious and asked by many question in our mentorship's first week meeting. The Linux kernel is huge ! It is very easy to get lost reading the code and deciding what to do. The best way to start contributing to the kernel is - "Fixing the spelling mistakes". You read it right ! Fixing the spelling mistake should be your first patch. The kernel code, even today has many spelling mistakes. You can use tools such as spellcheck to find those. Fix the spelling, create the patch, just send it upstream and enjoy a small win. One might think, why there are still spelling mistakes in the kernel ? Why nobody fixed all of them at once as it is very easy to do so? I also amused about the same. I do not have an answer for this, but here is what I think - These spelling mistakes are intentionally not fixed ! Why ? because they are there to be fixed by new upcoming kernel contributor like you ! To give you an opportunity for your first patch, to help you get started with the kernel development. This first spelling mistake fix patch will give you a bit of confidence and help you understand how patches are created, submitted, reviewed and merged. Hence, even though this patch is not improving any kernel functionality, it is definitely helping you to understand the development process and flows. But remember, just 1-2 patches on this stuff, leave other spellings mistakes for other new people to fix. Do not try to fix 100s of spellings.

    Okay, now you have your first contribution, what next ? Below are few areas/paths one can choose: 

• Fix kernel coding style issues: Use checkpatch.pl script to find out various coding style issues and fix them. There is good amount of work needed in drivers/staging around this domain. Do try your hands on that.
• Coccinelle tool: There are many coccinelle scripts under scripts/coccinelle. Run them, understand what each script is finding, develop the fix, and post it for review. Briefly, coccinelle is a code pattern finding tool which can search problematic code patterns. Many of my successful patches that I submitted upstream came from this tool.
• Static code analysis tools: There are many static code analysis tools like sparse, coverity, gcc compiler, smatch etc. You can attempt to fix warnings/issues reported by them. However, please make sure that the issues reported by these tools are genuine and not false-positive. As these tools tend to give many false positive because they do not have full context of the software design. Hence, while submitting fixes pointed out by these tools, make sure the problem really exists and your code is fixing it.
• Dynamic code analysis: Give a try on kernel syzbot reported issues. The key point here it ability to reproduce the syzbot reported bugs and then fix it. 

    Okay, now you got enough area to explore for contribution opportunities. From this point onwards, I think journey for each contributor deviates based on their interests. The linux kernel has many subsystems - core kernel, drivers, memory management, testing, networking and list goes on. You simply can not be expert of everything. You need to choose the subsystem and stick to it for a while untill you develop enough understanding about it and contribute well. Hence, I suggest each one of the readers to investigate what interests you. 

    As a beginner, I would recommend to choose a subsystem which has very active community and supportive maintainers. Spend a day or two browsing the kernel mailing list to find out which subsystems have active maintainers/reviewers. Working on fast moving and dynamic subsystem boosts your confidence as you will actively get feedback/comments on your patches on time. Find subsystems with maintainers and reviewers who are very encouraging and welcoming to new developers. As per my experience, each subsystem has their own pace and style of execution. So as a newcomer you need to find relatively faster and welcoming subsystems, do spend time finding those. In my experience and research during this mentorship program I found iio, hwmon, phy, kselftest, tee drivers, spacemit SoC and drivers/staging as fast moving and very welcoming. One can work on those if it interests you.

Make lkml your best friend:

    The linux kernel mailing list (LKML) is the most important place to understand and keep pace with kernel upstream development. Honestly, you need not to read any book, watch youtube videos, or take formal training to learn how to write good patches. Reading mailing list patches is the best way to understand how to write patches for particular subsystem. Coding style and mechanics of each linux subsystem might be slightly different. Reading mailing list daily will give you a good understanding about how development happens in a particular subsystem. You will also get to know what kind of contributions are welcomed and which are rejected on the subsystem. There is a lot to learn from lkml indeed. Mailing list discussions helps you to understand why certain decisions are made in the code and reasons behind it. This helps you to create your own patches. In fact, while reading the kernel code, do git blame at the code and check relevant commits/patches on the mailing lists, explore how that change is discussed, cross questioned, improved and ultimately merged in the mainline linux kernel. When you are fixing something or developing a new feature, do check out the mailing list for similar patches to avoid repeated submissions. There is search syntax used on lore.kernel.org which helps us to search patches, learn it. Keep observing subsystem mailing lists and be in touch with what are the hot things happening or design decisions being made. I typically spend minimum ~15 mins of my time reading the mailing list daily. It feels like reading a daily newspaper now a days !! 

Building trust with reviewers and maintainers:

    Here comes the magic of interpersonal skills and communication. As kernel development happens remotely and over emails. You are most probably not working with reviewers and maintainers in person. Hence, building trust between people is very important in kernel development according to me. I came to know from one of the mentorship sessions is that, there is deep meaning behind singed-of-by tag that you give in your commit/patch. The singed-of-by tag is not just the string that you attach or your git commit -s command attaches to your commit, it has a rather deeper meaning. It implicitly says that you have owned that change, you have fully tested that change on all possible hardware you have, the change you are making is correct and does not break the kernel with best of your knowledge. This tags adds originality to your code, it tells maintainers to trust your code. Hence, when you add singed-of-by tag, remember that you are implicitly telling that the code is trusted and you own it.

    Okay, switching gears to another domain of building trust. When we develop/debug kernel code, we put efforts to do so, and remember, equal effort is put by maintainers and reviewers to review and merge your code. Review does not come for free, it involves time and expertise from the maintainers. Hence, do appreciate the time and efforts spent by the reviewers on your patch and do not take review comments for granted. Be polite and showcase appreciation on the mailing list for the feedback provided by the maintainers. In my opinion, every commit in the kernel has 50-50 credits to author and reviewers. You will make sure that things work at your end, but reviewers make sure that they work in long run without any hidden bugs, hence appreciate their time and give them credit. This is what makes linux kernel's nature of development very successful. This approach helps to build the trust.

    One last point on building the trust. Lets say you submitted a patch, you got 5 comments as a feedback from the maintainer. I suggest to reply and ack on every comment and mention - how will you fix this in the next version of the patch, why will you fix this, what is the positive impact this change will have in the next version. This approach will help to communicate that you understood their comment/feedback and taking correct actions after understanding change fully. This definitely builds trust between author and maintainers. 

Writing code is just 50% work done:

    I want to start this section with an example. During this mentorship program, I worked on adding new driver for pressure sensor. Here is the story - It took me 2 weeks to setup the hardware, buying sensor chip, soldering, figuring out how to connect it to my single board computer, figure out device tree stuff, reading and understanding sensor datasheet, and finally writing and debugging the driver code. Wow, in 2 weeks I had a brand new driver working and ready to rock on the mailing list !! I took a deep breath and though I am done, Now I will send this patch series upstream and I will just get it merged, went to bed happily. Next morning woke up to find 13 comments on my patch !! Wohh, Then I realized, you are writing the code for the upstream kernel, standards are very high here. As a developer, you will always feel what you write is correct - in more standard way, it is called "developer biases". Hence we need to consider the fact that as a developer we will make mistakes, our code may not be optimal. Once you write the code, it is only 50% work done, next 50% work starts from review. Take all review comments and feedback positively and constructively, work on them and create new version of the patch. This iteration will go on till both, you and maintainers/reviewers are okay with it. And then work is 100% complete to be merged on mainline kernel. Okay, so in my case, patch that I was thinking that would get merged soon, took 5 revisions ! and 4 weeks. And I am happy with this, because I got to learn a lot of things during review phase which I can not forget now. Hence, moral of this section is - do expect a lot of good review/comments on your patch version and be open to accept and fix them. Really, kernel development is very structured process and you need to be very patient and consistent to tackle this. 

How to leverage this mentorship program:

    This mentorship program is fully self driven. You need to find problems to fix, you need to solve them, send patches upstream, follow up with the community of reviewers and maintainers, get it merged in mainline kernel. The best way to take help from mentors is asking right questions at the right time. Typically, we meet once in the week and discuss questions and queries that you have. Please be open and ask any dummest question about kernel development process you have. There are certain unwritten rules that kernel community follows. Try to understand them by talking to mentors during these meetings. You can ask questions about your specific patch. You can ask for feedback for your patch. Questions are limitless, hence do ask them without hesitation to get most out of this program. I used to maintain my personal doc with me where I would list down my questions as and when I get them, so that I do not forget them over a week. Hence, on the day of your meeting, you will have your questions ready. Seriously, this helped me a lot. Because if we do not note them down, we tend to forget them and miss the golden opportunity to ask them to mentors during the meetings. You can also verify your style of working or style of patch writing from mentors by asking feedback, this will give you more confidence. 

Hardware setup for embedded linux:

    When you aim to contribute to the drivers for sensors like pressure, temperature, humidity, proximity, voltage etc. It is very important to have a test setup to test your code. Most of these sensors interface over I2C or SPI. Hence, we need a single board computer(SBC) running linux to interface with those and test the drivers you write. Today, there are many single board computers in the market like beaglebone, rasprerry pi, banana pie etc. Now question is how to select out of those. As per my preference one can consider few points before selecting the board. Select the SBC which has very active upstream support and community. Select the board which supports build systems like buildroot/yocto. This will help us to build the software directly with upstream code. One more advantage of this approach is if your board has upstream support, you can run tip of the mainline kernel tree on your board without much of a hustle. For example, I use TI AM62x SK board, which has buildroot support, so I can download latest mainline kernel tree, build it and boot it on the board without an issue. This allows me to work directly on upstream mainline kernel without relaying on vendor provided custom kernel tree. I believe other options include Raspberry pi, beaglebone, and newly launched arduino Q. Another useful point while choosing SBC is number of IO ports it has. Choose one with more IO ports with standard headers. You should be able to use breadboard jumper wires to connect to it in case you have your sensor circuit on the breadboard.

    Once you have the SBC, read the user manual and schematics. Try to understand the various ports/pins on the board, understand the device tree for the board, compile the basic image and try to boot it. I use buildroot to build my images, where I direct buildroot to build my custom cloned and modified kernel using local.mk file. Next, you need I2C/SPI based sensors to interface and write/debug drivers for. Search kernel repository and check which sensors are currently supported. Try to find if there are any TODO's related to missing functionalities in any of the drivers. Try to find if there is any new sensor hardware for which there is no driver in the mainline kernel. Finding a new sensor to write a driver for is indeed a challenging task. Try to browse through various electronic component websites and look for new hardware/sensor release. One more important tool I would like to mention is logic analyzer. Get yourself a logic analyzer (minimum 24MHz) if you do not have one. These days they are not much expensive, you can get basic one for ~25$.  It will help you to see I2C or SPI transactions on the bus. This helps a lot while debugging your I2C/SPI client driver that you write for sensors. Spend time analyzing signals and understand how host and sensor communicates over the signals.

    During my mentorship program, I worked on adding new features on rtc chips, new driver for pressure sensor in the iio subsystem. Typically, most of the new sensors chips may not have module available in the market. Hence, learn soldering. I bought couple of rtc chips and pressure sensor chips, soldered them on breakout boards and created my own modules ! It was fun working with electronics stuff.

    One last point working on sensor drivers. Device tree bindings are very important. If you are writing any new driver, you need to write device tree binding documentation for it. Kernel community has very strict rules on how to write those bindings. Hence, do spend good amount of time understanding them. Device tree binding describes the hardware, like how many interrupt outputs, what is the interface, max frequency, i2c address etc. Definitely, one can find good amount of information about it on the web. 

Concluding remarks about kernel mentorship journey: 

    Overall, I had a very positive experience with this mentorship program. There are many online resources teaching about how to contribute to the upstream kernel. However, what these resources lack is the personal doubts or questions one may get during this journey. This program solves exactly this problem. We have a lot of information about the kernel and upstreaming online and in books, but the challenge is how to consume it in the most effective way so that it goes into your blood. This program has a unique structure of doing and learning on your own and getting doubts cleared from the mentors during the weekly meetings. I think the place where this program helped me is keeping myself on track and resolving my doubts and confusion about the community practices. I experienced that upstream kernel development is 50% code and 50% engaging with the community. The kernel community has few unwritten rules, this program helped me to understand those small-small things which we never see in formal documentations or books. This program helped me to gain confidence about my patch writing skills by weekly guidance from upstreaming experts. Hence, I feel this is the unforgettable and one of the valuable experiences in my linux kernel journey. And lastly, this journey taught me - How to be patient and persistent with your patches. Looking forward to apply these skills and keep contributing to the biggest and the most important software project on the earth.

References and Links: 

[1] https://wiki.linuxfoundation.org/lkmp

[2] https://wiki.linuxfoundation.org/lkmp/lkmp_mentee_blogs