2023-09-25T12:57:35+00:00http://siteshwar.github.com/OpenScanHub2023-08-11T00:00:00+00:00http://siteshwar.github.com//posts/openscanhub<p>OpenScanHub is a service for static and dynamic code analysis. It was internally used inside Red Hat for more than 12 years and was recently open sourced. This blog post is a brief summary about it.</p>
<p>OpenScanHub is primarily used to scan RPM packages and source tarballs. It is extensible through <a href="https://github.com/csutils/csmock">csmock</a> plugins and is not limited to any programming language. It uses open source tools like <code class="language-plaintext highlighter-rouge">Cppcheck</code>, <code class="language-plaintext highlighter-rouge">ShellCheck</code>, <code class="language-plaintext highlighter-rouge">gcc</code>, <code class="language-plaintext highlighter-rouge">clang</code> etc. to find defects in the code, but it is also extensible to proprietary tools like Coverity and Snyk. In simple words, it is just a “frontend” to run various analyzers on a codebase. It is a service and does not have a static analyzer of its own. It can also perform dynamic analysis through tools like <code class="language-plaintext highlighter-rouge">valgrind</code> and <code class="language-plaintext highlighter-rouge">strace</code>.</p>
<p>It can collect all the reports from various analyzers at a single place. This makes it easy for the developers to track bugs in their code, as the reports are not fragmented across different places.</p>
<p>The most unique feature of OpenScanHub is the ability to perform differential scans. It can compare reports from a newer version of a codebase with the older version and can report defects that were introduced in the newer version. This is an important feature while scanning legacy codebases as the maintainers do not have time to fix all the defects that may have accumulated over time. It can help them to avoid introducing new defects in the code. It is also an important feature for package maintainers (who are not upstream developers), as they can focus on defects that were introduced by a new release or a downstream patch.</p>
<p>RHEL 5 was the first RHEL distribution that was scanned through it. And it has been used since RHEL 7 to perform mass scans before new releases of RHEL. The most recent release that was scanned through OpenScanHub is RHEL 9. In RHEL 9, it analyzed 480 million lines of code in 3700 packages, where it identified approximately 680000 potential bugs. This helped us to eliminate hundreds of potential security issues during the productization phase of RHEL 9. It has become a critical part of RHEL releases and its usage is slowly growing with other projects.</p>
<p>I was responsible for open sourcing this codebase and I hope the entire open source community would benefit from it. I would like to thank <a href="https://github.com/kdudka">Kamil Dudka</a>, <a href="https://github.com/lzaoral">Lukáš Zaoral</a> and all the OpenScanHub contributors inside Red Hat that have supported this effort. This is the beginning of building a community around OpenScanHub and you can join the effort on <a href="https://github.com/openscanhub/openscanhub">GitHub</a>.</p>
<p>Comments on <a href="https://news.ycombinator.com/item?id=37090555">Hacker News</a> <del>and <a href="https://www.reddit.com/r/linux/comments/15odj7v/openscanhub_static_and_dynamic_analysis_as_a/">reddit</a></del>.</p>
<p>EDIT: My post at reddit was removed because of ‘Not Relevant to r/linux or low effort’. In future, I may refrain from posting to reddit.</p>
CentOS Stream - sync2git2020-10-21T00:00:00+00:00http://siteshwar.github.com//posts/sync2git<p>TLDR: This blog post summarizes <code class="language-plaintext highlighter-rouge">sync2git</code> service that is used to push internal Red Hat packages and modules into CentOS git repositories. I will skip talking about modules for simplicity.</p>
<h2 id="what-is-sync2git-">What is sync2git ?</h2>
<p>It’s a service that’s part of <a href="https://www.centos.org/centos-stream/">CentOS Stream</a> project. In simple words, it does 2 things:</p>
<ol>
<li>Pull internal package and module builds from brew. Brew is internal Red Hat instance of
<a href="https://koji.fedoraproject.org/">koji</a>.</li>
<li>Push them to <a href="https://git.centos.org/">CentOS git</a> repositories.</li>
</ol>
<h2 id="how-does-it-work-">How does it work ?</h2>
<p>It heaviliy depends on <a href="https://github.com/release-engineering/alt-src">alt-src</a> tool which is maintained by Red Hat release engineering team. <code class="language-plaintext highlighter-rouge">sync2git</code> works in several steps:</p>
<ol>
<li>Pull list of internal package builds that are tagged with certain tag. For example, <code class="language-plaintext highlighter-rouge">rhel-8.2.0-candidate</code> is the internal tag for builds that are set to be released for <code class="language-plaintext highlighter-rouge">rhel-8.2.0</code>.</li>
<li>Check if there are any embargoed CVE fixes pending for any of the listed builds. If yes, remove them from list. We want to avoid publishing any packages to git.centos.org that contain CVE fixes that are not released yet. So, package builds that contain unpublished CVE fixes may take longer than other packages to be available for CentOS Stream. There is an internal hosted web service to check for such packages.</li>
<li>Download source rpm for the listed packages.</li>
<li>Use <code class="language-plaintext highlighter-rouge">alt-src</code> to push the source rpm to git.centos.org. <code class="language-plaintext highlighter-rouge">alt-src</code> basically explodes the source rpm and pushes the sources to <code class="language-plaintext highlighter-rouge">c8s</code> branch in git.centos.org. For example, latest changes for systemd can be seen <a href="https://git.centos.org/rpms/systemd/commits/c8s">here</a>.</li>
</ol>
<h2 id="where-can-i-find-the-details-">Where can I find the details ?</h2>
<p>A line of code is worth a thousand words. Some of the details I shared in previous section may change over time. For details and latest updates you can see code for this service <a href="https://github.com/CentOS/sync2git">here</a>.</p>
<h2 id="credits">Credits</h2>
<p>Thanks to <a href="https://github.com/bstinsonmhk">Brian Stinson</a>, <a href="https://github.com/james-antill">James Antill</a> and rest of CentOS Stream team for helping me with implementation of this service. Also, thanks to Red Hat release engineering team for their guidance with <code class="language-plaintext highlighter-rouge">alt-src</code>.</p>
<p>Comments on <a href="https://news.ycombinator.com/item?id=24848678">Hacker News</a> and <a href="https://www.reddit.com/r/CentOS/comments/jfevrw/centos_stream_sync2git/">reddit</a>.</p>
KornShell 2020 - Impossible Happens!2019-10-10T00:00:00+00:00http://siteshwar.github.com//posts/kornshell-2020-impossible-happens<p>TLDR: Christmas came early. ksh-2020.0.0 was released today. Read the release announcement <a href="https://groups.google.com/d/msg/korn-shell/-tQkll8BxXU/X4ON61CHBwAJ">here</a>.</p>
<p>More than 2 years ago, I <a href="korn-shell-not-dead">announced my intentions</a> to revive AT&T KornShell. When I started working on it, this codebase was a big hairball that nobody wanted to touch. It’s written in such a tricky way that can make the best C programmers sweat. Build system was complicated and debugging build failures was a nightmare. Test coverage was bad and even a simple bug fix could end up breaking basic functionality. There were lots of old bugs that have not been fixed for decades. It seemed almost impossible that we would be able to make a new release.</p>
<p>Today we are releasing stable version of ksh-2020.0.0. This marks the first stable release of ksh after a gap of more than 6 years. I will summarize key changes here:</p>
<ul>
<li>
<p>More than 500,000 lines of code were dropped to simplify the codebase.</p>
</li>
<li>
<p>Build system was changed from Nmake to Meson. This improved build speed by a factor of 35 and made it easier to debug build failures.</p>
</li>
<li>
<p>Downstream patches from different vendors were upstreamed.</p>
</li>
<li>
<p>Hundreds of new tests were added to get better test coverage. This includes writing interactive tests for vi and emacs bindings through <code class="language-plaintext highlighter-rouge">expect</code> based framework.</p>
</li>
<li>
<p>Coverity defect rate (bugs per thousand lines of code) was brought down from more than 2 to less than 0.5. This is below average defect rate of projects that are scanned through Coverity.</p>
</li>
<li>
<p>Refactored thousands of lines of code to make the style consistent and easier to maintain.</p>
</li>
</ul>
<p>These changes have helped to bring this codebase closer to 21st century and sparked a new interest in keeping ksh alive.</p>
<p>On a personal note, it was amazing to work with awesomest <a href="https://github.com/ridiculousfish">ridiculous_fish</a> to <a href="https://www.mail-archive.com/fish-users@lists.sourceforge.net/msg03044.html">revive fish shell back in 2012</a>. fish shell has come a long way since then and has managed to expand its community from very few people to hundreds of new contributors and thousands of new users. It’s a privilege to work with another gifted programmer, <a href="https://github.com/krader1961">Kurtis Rader</a>, to give a new life to ksh. Kurtis used to be a fish shell contributor, but I never got a chance to work together with him until he started contributing to ksh. It would not have been possible to make a release without his consistent hard work and enthusiasm. Also, thanks to <a href="https://github.com/kdudka">Kamil Dudka</a> for helping me in making key decisions and reviewing pull requests.</p>
<p>It was a big setback to ksh when David Korn and his team stopped working on it a few years ago. 10th October 2013 was <a href="https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01462.html">the last day</a> of David and his team at Bell Labs. As a tribute to previous maintainers, we are making this release on 10th October 2019. We may not be able to bring long term vision previous maintainers had, but we can try to keep ksh relevant to the present time. I am hoping this release will open new frontiers for the future of KornShell.</p>
<p>It’s not the end, it’s a new beginning!</p>
<p>EDIT: I stopped working on ksh after AT&T decided to <a href="https://github.com/att/ast/issues/1466">rewind</a> the repository back to it’s old state. I summarized my view of this project <a href="https://github.com/att/ast/issues/1464#issuecomment-581458320">here</a>.</p>
<p>Comments on <a href="https://news.ycombinator.com/item?id=21214334">Hacker News</a> and <a href="https://www.reddit.com/r/unix/comments/dfy6lp/kornshell_2020_impossible_happens/">reddit</a>.</p>
Squeezing Water from Stone - KornShell in 20192019-02-11T00:00:00+00:00http://siteshwar.github.com//posts/kornshell-in-2019<p>At FOSDEM 2019 I gave a brief status update about state of AT&T KornShell. <a href="https://fosdem.org/2019/schedule/event/kornshell/">Video</a> of my talk is now available at FOSDEM website. I mostly talked about what makes ksh so challenging to maintain and about the steps current upstream maintainers have taken to revive it. If you want to provide feedback, feel free to drop me an e-mail or give me a shout on Twitter.</p>
<p>Edit: I gave an updated version of this talk at <a href="https://www.youtube.com/watch?v=9J20eGwGCwU">All Systems Go! 2019</a>.</p>
KornShell - Moving forward2017-12-04T00:00:00+00:00http://siteshwar.github.com//posts/korn-shell-moving-forward<p>Since my <a href="http://situ.im/posts/korn-shell-not-dead">last blog post</a> we have made some decent progress to improve code around ksh93. I will try to summarize recent developments here :</p>
<ul>
<li>
<p>Legacy build system has been deprecated and we are going to move to <a href="http://mesonbuild.com/">Meson</a>. The only part of legacy build system that still remains is use of ‘iffe’ (if feature exists) command. It is used for feature detection (same like configure script in Autotools) and will require some extra amount of work for replacement.</p>
</li>
<li>
<p>Travis integration has been improved. Now we are running test cases on Travis to detect regressions. The new build system allows us to run test cases in parallel. With the legacy build system, building and testing used to take more than 20 minutes, it has been cut down to around 5 minutes now.</p>
</li>
<li>
<p>We have moved to ‘master’ branch for development. Most of the git projects use ‘master’ for development. But since <a href="https://github.com/att/ast">AST repository</a> contained full source code of all AST projects, it took us some time to reach a conclusion on it. At the end, we decided to move full AST source code under separate branches and use ‘master’ only for ksh93 development.</p>
</li>
</ul>
<p>I want to say thanks to my fellow committer <a href="https://github.com/krader1961">Kurtis Rader</a> for his invaluable contributions. He has been instrumental in improving code and has agitated a few discussions in the community. There is a growing interest in the community to keep ksh93 relevant. We are up to a good start and the list of <a href="https://github.com/att/ast/pulls">pull requests</a> and <a href="https://github.com/att/ast/issues">issues</a> is slowly increasing. It is a sign of recovery. KornShell is heading for good times!</p>
<p>Comments on <a href="https://news.ycombinator.com/item?id=15862844">Hacker News</a> and <a href="https://www.reddit.com/r/unix/comments/7hzhna/kornshell_moving_forward/">reddit</a>.</p>
KornShell is not dead2017-07-01T00:00:00+00:00http://siteshwar.github.com//posts/korn-shell-not-dead<p>Despite the lack of pace in development from last couple of years, ksh still remains one of the most popular shells. AT&T released source code on <a href="https://github.com/att/ast">GitHub</a>, but there were no updates from them for more than an year. Last commit was made by AT&T on <a href="https://github.com/att/ast/commit/c506cb548d9b4bcebef92c86e948657728760e15">Jan 11, 2016</a>. I have been submitting bug fixes on the mailing list, but the patches were not added to the upstream repository as there is nobody working on KornShell at AT&T. This week they provided me commit access to the GitHub repository. The latest code was released under beta branch and I am going to build on top of it. I made several fixes this week and I am planning to do more in coming days.</p>
<p>There is still a decent user community around KornShell and they want to keep it well maintained. I look forward to work with them. Let’s see what we can build together!</p>
Sailfish OS for Nexus 5 (alpha) is out2014-08-08T00:00:00+00:00http://siteshwar.github.com//posts/sailfish-os-nexus5-alpha1<p>Today we are releasing first image of Sailfish OS for Nexus 5. This is the first CyanogenMod 11 based port of Sailfish OS and it has been pretty stable in our tests.
Instructions to download and install the image are available from <a href="https://wiki.merproject.org/wiki/Adaptations/libhybris/Install_SailfishOS_for_hammerhead">wiki</a>. Big thanks to <a href="https://twitter.com/vgrade">vgrade</a>, <a href="https://twitter.com/stskeeps">stskeeps</a>, <a href="https://twitter.com/lbt_">lbt</a>, <a href="https://twitter.com/sledgesim">sledges</a>, all the Jolla sailors and awesome people from community for their contributions in this release. Join us on <a href="http://webchat.freenode.net/?channels=#sailfishos-porters">#sailfishos-porters</a> to discuss more about this port.</p>
Hello World2014-03-21T00:00:00+00:00http://siteshwar.github.com//posts/hello-world<p>Hi! I am an engineer at Red Hat and mostly work on GNOME/KDE. I am also a maintainer of fish shell and contribute to several other Open Source projects. <del>My IRC nick is situ@freenode/oftc.</del> I hope to use this place to share some useful stuff. Stay tuned !</p>