Scan 2009: Security for Open Source Software

Samba’s Jeremy Allison: Take a Cue from Open Source and Drop Some Ego

July 21st, 2009 Post a comment »

Jeremy Allison is co-creator of Samba (the other being Andrew Tridgell), and has been working on the project since its inception 16 years ago. Allison is also the Scan Samba Project Leader. He continues our discussion of open source vs. proprietary and speaks on why there can be benefits at the intersection of the two, even though he thinks open source will eventually throttle proprietary development.

Q. How is Samba used and what’s the appeal of coding in it?

A. Samba code is used to provide Windows compatible file, print and authentication services to UNIX or Linux based platforms. It’s a standard component in most Linux distributions, and is shipped commercially by many of the UNIX distributors (Sun, Apple, IBM, HP).

Samba is also widely used in the embedded NAS (Network Attached Storage) space, powering a wide range of NAS devices ranging from the small single disk devices to multi-petabyte clustered storage arrays.

Samba is written in simple C, with no parts in more exotic languages. This is a strength for competent developers who understand the strengths and weaknesses of C, but can be off putting to programmers who expect the language to take care of memory management and garbage collection.

Q. What are some of the ways Samba has progressed as a code base?

A. Starting from a simple UNIX “network daemon” code base, Samba has developed several libraries that are incredibly useful spin off technologies and products in their own right, such as talloc (hierarchical memory allocator), tdb (trivial database - memory mapped multi-simultaneous reader/writer database) and ctdb, a clustered version of the tdb code.

Samba is now developed in a much more rigorous way than before, with mandatory test suites for new features to prevent regressions, and a full test suite that even Microsoft uses to test their own Windows server code base.

Q. With wide deployment across major commercial vendors like Sun, Apple, IBM, and HP, how does Samba manage security measures?

A. Samba is a long-lived network service, which has to protect itself against malicious code and denial of service attacks.

As Samba ships as part of so many products, it has been audited heavily from both a security and legal perspective, by experts in these fields (the Linux and UNIX distribution security teams, for example). We have a dedicated security email address with 24×7 coverage, and have produced fixes with a turn-around time of one hour on occasion. We work closely with the “vendor-sec” security list to ensure our code is as secure as possible. On the whole, the Samba community has always been concerned with security and quality, and the code was originally designed with this in mind.

Q. Is a software integrity issue different from a security issue?

A. Total software integrity is a vital part of security. Known bugs can sometimes turn into security issues if they’re not correctly understood or addressed. 100% bug-free, reliable software is our goal, and one that Coverity scans play an important part in achieving.

The biggest code quality issue someone can have is lack of testing. We’re attempting to get 100% test coverage of our code, and our motto has slowly become “Untested code is broken code.” Within the Samba community, we’ve started a pair-peer review process for all check-ins into the tree to increase accountability and robustness.

Q. Getting into the debate over open vs. proprietary, many arguments have been made for one or the other. What’s the real advantage to the open source model (if one exists)?

It’s an old argument, but the “many eyes” of an active community do make a difference. People study areas of Samba that are not under active development and are considered “stable” and point out the most minor errors in order to improve Samba, giving Samba and the person who found the error more credibility, which I’m always happy about.

In my opinion, proprietary software development groups can take a cue from what the open source communities have done with code integrity. Opening your code is the best thing you can possibly do for code integrity. I’ve seen many proprietary developers be very resistant to this as they’re scared of what external eyes might find. Yes, it will be painful at first, but it’s a major increase in code reliability. You end up with much less of an ego, as what you think is perfect code gets pounded with bug reports that people can prove are errors, day after day. It helps achieve the programming nirvana of “egoless programming.”

Q. Have you seen overwhelming benefits of open source software with commercial adoption, and if so, has that commercial aspect shifted the pressures developers face in terms of releasing secure, quality code?

Commercial software development can afford better tools than FLOSS developers usually. We’re very grateful for the free coverage we get from Coverity and the Scan Program. As a project, we would not have been able to afford the tool ourselves.

Also, commercial software should go open source whenever it can. It really is the scientific method as applied to software development, and it really is the better way to develop software.

Pure commercial code developers face more pressure to just “ship with what they have,” without taking the time to cover all outstanding bugs. FLOSS developers can ignore that pressure, but without a hard deadline it’s easy to get caught in “perpetual rewrite” mode, without ever shipping anything. This is where a good release and community manager can make all the difference.

In time, more and more code will become FLOSS. Eventually proprietary will die
out - just like the alchemists slowly turned into chemists and materials scientists. In the future, people will expect access to code in the same way they expect blueprints to a new building or bridge.

Q. What has Samba gained from Scan?

I’m very involved with Scan, and I usually start my week reviewing the Scan output. On the whole, projects need to pay attention and fix the bugs that Scan reveals. If you’re getting false positives, it’s usually worth it to try and rewrite the code to avoid them, as the real errors found by Scan can save the reputation of a project.

Would you like to know about 0day defects months in advance?

July 20th, 2009 Post a comment »

There’s a lot of discussion today about a 0day Local Linux Root exploit. http://isc.sans.org/diary.html?storyid=6820 For readers who aren’t security-savvy, that means that a user logged into a Linux system, with shell access, can bypass system security mechanisms and elevate his access to be equivalent to the system administrator (or ‘root’ user).

It’s called a 0day because the exploit was released with no advance warning that people should patch their systems - even though the code change to close the hole was committed twelve days ago, on July 5th, lots of people still have systems that are running on vulnerable versions of the code.

As part of the Scan Project, we’ve been analyzing a wide range of open source projects. We generally don’t talk about the specifics of the issues we find, and when we do, we wait until long after patches have been issued, and security advisories published. 0day defects are very disruptive for system administrators, since they demand immediate attention - the consequences of ignoring a 0day can lead to spending more time to completely reinstall a system when someone uses a 0day and then installs a rootkit (persistent backdoor access).

Generally, creating a specific exploit involves three things:

Knowledge of a particular technique for bypassing the protective mechanisms that are built into a system
Finding source fragments with examples of the code construct required by the technique.
Locating a point on the attack surface which lets you provided the needed trigger, and can reach the targeted vulnerable code.

You can think of this like inventing a Phillips screwdriver, locating some screws with a compatible head design, and having the screws be somewhere you can reach with the screwdriver. If any one of the three factors isn’t there, you can’t accomplish your goal. Of course, there’s more than one type of screwdriver, and there’s more than one type of exploit technique. All you need to find is one matched set of three.

In the case of today’s drivers/net/tun.c 0day, the relevant code fragment is this:

static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
  	{
  		struct tun_file *tfile = file->private_data;
  		struct tun_struct *tun = __tun_get(tfile);
  		struct sock *sk = tun->sk;
  		unsigned int mask = 0;

  		if (!tun)
  			return POLLERR;

The actual exploit is complex. It involves a compiler optimization that prevents this code from working as written, and a mechanism for mapping the zero-page of memory, so that a NULL pointer references memory under the control of the attacker. Finally, the attacker needs execution to continue past the safety check, which would normally return the POLLERR error code.Others have done a good job of covering the details, so I’ll provide URLs to their write-ups below.

The code was added to the Linux kernel on Feb 5th, 2009. Specifically, the ’struct sock *sk’ line was added. Some of the comments on the URLs below include remarks such as “a source code audit of the vulnerable code would never find this vulnerability”, and “That’s still perfectly valid C code”.

http://www.reddit.com/r/programming/comments/921sg/root_hole_in_linux_2630_including_a_creative_new/

On the face of it, this appears to be a simple null pointer defect. If the ‘tun’ pointer returned from __tun_get(tfile) is NULL, then the expression tun->sk should cause a null pointer dereference. For programmers used to working outside of the kernel, the usual result of dereferencing a null pointer is a segmentation violation, and the operating system terminating the application process. When programming in kernel space the result is specific to your platform and environment.

The unusual situation in this particular defect is that the misbehavior happened at compile-time. gcc looked at the assignment to sk, which used tun as if it was guaranteed to be a valid non-NULL pointer, and decided that there was no point in performing the if (!tun) test below. The if() block was optimized out. It’s in the source code, but it is NOT in the machine code that the compiler output.

In addition to fixing the code block in question, changes were also committed to Linux on July 16^th to disable this particular compiler optimization during the build process.

So, this is clearly not expected behavior on the part of the compiler. The programmers included a test to check whether tun was valid or not, and return an error if it was not. The compiler removed that test, resulting in binaries that would allow processing to continue past the intended checkpoint, even when tun’s value was invalid.

Now, let us go back to the comment that “a source code audit of the vulnerable code would never find this vulnerability”. It certainly true that manual auditing would not be likely to predict that the compiler would interpret the code in such a way that the explicit test of tun’s validity would be bypassed. However, code auditors might very well have noticed the contradiction in dereferencing a pointer before checking to see if it is NULL.

There is a further challenge though. Even if the code auditors in question are well versed in every aspect of the programming language they’re reviewing, keeping in mind every best practice and comparing them to a set of lines of code is very difficult. It’s difficult even when the lines are close together such as in this example, and it approaches impossibility when the lines are further apart, or separated by indirection, like macro definitions, and function calls. It also gets harder when there are multiple paths through the code, and every possible combination has to be considered.

Fortunately, there are tools that can identify code issues just like today’s vulnerability. In fact, there’s at least one tool that identified this particular issue specifically. The Scan Project analyzes the Linux kernel on an ongoing basis. This issue was identified months ago, and tracked up until the fix was committed. For developers with access to the Scan results, it is Coverity defect ID #13020. As it happens, our Linux builds were not running smoothly in February and March, so we didn’t analyze a version of code containing the change until March 31st. At that point, the analysis flagged the issue as an instance of ‘Use of NULL value before test’, and continued to find it until July 6th when it had been fixed.

Now, note that in the vulnerability revealed today, the questionable optimization behavior in gcc predated the introduction of the code in the Linux kernel. However, it could as easily have happened the other way around - Linux adding the code first, and then developing a kernel vulnerability when people started using newer versions of the compiler. That’s why it’s important to maintain good coding practices. Whether it’s a change in the compiler, or the platform, or a change introduced by another developer working in your code later, keeping the code clean of issues like this one will not only protect you from vulnerabilities today, but keep new ones from opening up in the future.

The Linux developers have been doing preventative maintenance using the Coverity Scan for some time. There are hundreds of issues that have been fixed in Linux as a result of being identified in the Scan, and some of those would have provided other avenues for today’s 0day exploit as well. We’re glad to see those fixed, and know that they aren’t still available to be used in exploits.

The author of the exploit (Brad Spengler of grsecurity.net) lists the timeline for his work. He starts by noting ‘Discovery time of bug in public: 7/6/09′. That time corresponds to the day after the bug had been patched in Linux, so we should be safe to assume that the discovery was made by reading the commit logs which included the reason for the fix.

The Scan analysis data is kept private, available only to the developers who are approved by our contacts within Linux’s leadership. Publishing the analysis results would give exploit authors the opportunity to identify target code like this one, and would be irresponsible of us. We thank the Linux developers for their ongoing efforts to harden the Linux codebase, striving for complete code integrity. We encourage the Linux developers who don’t yet take advantage of the Scan Project to help those who do, in their efforts to fix outstanding issues like the one behind today’s vulnerability, and to keep such issues out of the Linux codebase on a continuing basis.

http://isc.sans.org/diary.html?storyid=6820

http://lists.grok.org.uk/pipermail/full-disclosure/2009-July/069714.html

http://www.reddit.com/r/programming/comments/921sg/root_hole_in_linux_2630_including_a_creative_new/

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a3ca86aea507904148870946d599e07a340b39bf

Frank Kardel: Being “On-time” with Open Source Security, Part 2

July 14th, 2009 Post a comment »

In Part 2 of our Q&A with Frank Kardel, we find out how the NTP community handles code security and dive into the long-fought debate over the merits of opens source vs. proprietary software. Beyond that, Kardel addresses where the two can potentially learn from each other to improve software security.

Q. In your 21 years of experience with NTP, what’s been the biggest code quality concerns?

A. Several areas have room for improvements. For one, the NTP reference implementation runs on a very wide range of operating systems including Unix(like) environments (commercial and open source), Windows and proprietary operating systems. Supporting so many OSs remains a challenge due to limited test environments.

Second, the NTP reference implementation makes use of operating system interfaces that vary among different vendors and have different limitations. The challenge is finding a correct solution for a certain platform, which isn’t always easy. On the positive side however, deployment on many environments has helped to detect a number of oversights and inaccuracies in the code leading to an overall improvement and strengthening of the code base.

Q. Security seems to be a priority of NTP. How does NTP tackle security from a community manager level?

A. NTP has defined procedures to address security. Having a secure code base is critical to our success because implementations of NTP usually work silently along with other software, and it’s widely deployed by both commercial and non-commercial users.

Q. Would you say the strength of NTP’s code base has grown?

A. Over the decades, NTP developers have rewritten the code base several times in order to keep up with correctness, performance, portability and protocol evolution. The NTP code base has the most complete implementation of the protocol suite and is stronger than it’s ever been.

Q. You champion open source, but do you see any limitations to the open source model with respect to maintaining code integrity?

A. For one, NTP funding is limited with respect to development, so that can be a disadvantage of open source. On the other side, the NTP code base can evolve more freely without proprietary constraints. Thus, we can take our time and invest in not only in features, but also in code integrity and correctness.

If proprietary constraints were to become overwhelming, there would be a danger of sacrificing integrity and correctness.

Q. Could proprietary software teams learn something from how open source communities address integrity?

A. When interpreting code integrity as having manageable and correct design, the lesson could be that it pays off in the long run to prefer sound software architecture over focusing on solutions geared for short-term benefits.

Another lesson — sometimes it may also be worthwhile to do a fresh start. Re-implement software with good and proven components and get rid of cruft that has accumulated over the time.

Q. Finally, what best practices should the OSS community and proprietary vendors adopt from each other regarding quality and security?

A. In general both OSS and proprietary vendors could benefit from producing robust, correct software based on sound engineering practices. Open source software by itself is not better or worse, but it definitely can be more flexible for trying out new ideas, possibly at the dispense of code integrity.

Depending on the focus of a project, its deployment and the damages a failure could cause in either model, the driving factor behind security and quality is a necessity.

There are several areas where code integrity becomes an important factor. Ones that come to my mind are:

- With a high deployment base, efficient incident handling is necessary. Otherwise small mistakes can cause big failure impacts.
- With high risk applications, where the application could be used for attack, and can cause bad reputation and high damages.

Basically, it boils down to whether a project (OSS or proprietary) is able to support its product even when it is widely deployed or used in high risk environments. In order to be able to support such a product, it is very helpful to follow a design scheme that supports code integrity. Without that, it will become increasingly difficult to support that product and the competition can gain market share.

Frank Kardel: Being “On-time” with Open Source Security, Part 1

July 9th, 2009 Post a comment »

Frank Kardel is a veteran contributor to the Network Time Protocol (NTP).

Over the last 21 years, he has made significant contributions to maintain, update and secure the NTP code. Kardel is also the NTP Scan Project Leader.

In Part 1 of this Q&A, Kardel talks to us about the NTP community, how they’ve dealt with code security over the past two decades and how NTP prioritizes the demands of its code.

Q. Tell us about NTP and where we might see it in use?

A. NTP is one of the oldest Internet protocols and has been used in the Internet for over two decades now. Today, almost everything connected to the Internet has support for NTP, which solves the basic problem of obtaining a reasonably correct time in devices connected to the Internet. There are millions of devices out there (think computers, media center set top boxes and even small home routers).

Having correct time is useful for a normal user, but it is essential in the security arena when analyzing a sequence of events to find out what happened during a break in, for instance. Time is the only widely available means to correlate event traces.

A whole industry has developed around producing products that provide reliable time services. These devices usually combine highly stable and reliable reference time sources synchronized to official time dissemination services through a specialized microcomputer that handles the network protocol.

NTP solves the basic need of providing time synchronization to many Internet connected devices. As such, a market to provide easy to use turnkey solutions emerged for companies acting as local time servers. The demand for public time servers has grown tremendously over the years, leading to a creation of a public time service infrastructure project, which serves millions of clients. Current project estimates show NTP service requests between 40,000 and 120,000 per second, which gives us a small impression of how widely NTP is deployed today.

Q. So you’ve been involved with the NTP community for 21 years now. What’s your role today and how has that evolved during this time?

A. Between 1988 to 1992, I initially contributed a reference clock driver to the public NTP code base. Around that time I set up the first Stratum 1 NTP server at the University of Erlangen-Nürnberg.

The service (in German) I created is still available today, though with different hardware, and now serving at a rate of 11-17K requests per minute per server.

Over the last 21 years I have contributed several enhancements to NTP with respect to the cleanup and portability of the code base. Currently, I am mainly involved with the reference clock driver, the NTP I/O subsystem and Unix implementation issues.

Q. NTP is a relatively small OSS project. What is the appeal of contributing to NTP and participating in its community?

A. Looking at the contributor group, NTP is a very small compared to other OSS projects today like the Linux community. Dave Mills, the creator of the NTP specification and the reference implementation, likes to describe the project as a team of “more than 4 dozen contributors and countless bug catchers.” These volunteers come from several countries representing many professions and skills.

While it may seem that the NTP project ‘only’ provides a solution that ‘just’ solves distributing time, it is not as simple as it sounds at first.

We’re a very specialized area of development and the motivation for me at least is in providing a solid and correct reference implementation. What’s amazing is that even though it is a very small project, the impact of NTP on the Internet is tremendous. NTP has solved the time distribution problem on the Internet around the globe.

Q. For the Scan Project, you’re our official rep from NTP. As you know, Scan focuses on addressing performance, quality and security all concerns Coverity believes is essential for software integrity. What are your thoughts on these concerns and what they mean for the NTP community?

A. They’re all important, but not equal. The NTP implementation historically prioritizes “correctness” (even in rough environments). It’s about providing reliable time synchronization.

Security is a must due to elevated rights needed by the software for operation.

Performance was necessary in the days when the implementation was deployed on slow (by today’s standards) systems with limited memory.

Usually all three aspects are kept in mind when working on the reference implementation. However, out of the three, we can most easily make sacrifices in the performance (computational effort) area — especially if it improves security and/or correctness. An example here is the use of cryptographic algorithms that will take time to compute but increase security (trusted time servers) and correctness (resilience against attacks).

Ubuntu’s Jono Bacon: Open Source Code Priority Should Begin at Security, then Quality, then Performance

July 7th, 2009 5 comments »

Jono Bacon is the Ubuntu Community Manager and leads a small team at Canonical to grow and enable the Ubuntu community.

Q. The Ubuntu community has thousands of contributors, hundreds of teams and many diverse methods of contributing. What brings them all together?

A. Ubuntu is used in a variety of industries, from software to engineering to government to retail and more, and has become a general purpose Operating System that can offer competitive productivity and information management tools as well a wealth of specialized tools that are available for download.

It’s a definitive channel for software developers. Ubuntu has a solid desktop, a good server and mobile experience and an exceptionally large community that refines, extends and supports us. Ubuntu is one of the most popular Linux distributions available, and as such, is ideal for delivering applications.

Q. Ubuntu has become a global brand that is spoken in the same sentence of its well-established competitors. How has security and quality scaled as Ubuntu’s size has grown?

Ubuntu takes quality and security very seriously. We’ve seen extensive growth in the user base, customers and community contributors, and my role as community manager is there to ensure that our community scales as Ubuntu grows in other areas.

On one hand, we have definitive processes to manage security expectations including our Bug Triage best practice, our Stable Release Update policy, our ‘proposed’ updates and more. However, Ubuntu’s code security is not primarily managed by a community manager. It’s largely the domain of our Security Team and they have handled the expectations and responsibilities well.

In addition to this, we have been actively growing quality focused teams and refining how they collaborate with each other.

Q. Coverity’s philosophy of “software integrity” speaks of performance, quality and security, and was adopted into the Scan project to calibrate open source code. Do you agree that these are equally important for an open source code base?

A. I can’t really speak for Ubuntu as a whole, but speaking personally I would say that they are all critical, but priority should begin at security, then quality, then performance.

Q. In your opinion, is there an open source advantage when it comes to maintaining code integrity?

A. There’s a huge opportunity: it is the often spoken “more eyeballs on the code” philosophy. I think that is every bit as valuable today as it was when it was first spoken.

Q. What can proprietary software vendors learn from open source to advance code quality?

A. From a technical code quality perspective, proprietary software houses can learn a lot from the distributed open source model, such as how a community can refine the quality of its code when it has a solid onramp for contributions and a means of collaborating together effectively.

OSS and proprietary software will evolve in parallel, but proprietary software houses are going to need to embrace open source more as the industry demands it. Countering this idea, I do see more and more open source projects, platforms and products shipping commercial service opportunities for their users, so the growth prospects are exciting for both.

IBM Researcher and Postfix Maintainer, Wietse Venema on “Open” vs. “Closed” Source Code Security

July 2nd, 2009 Post a comment »

Wietse Venema created the open source Postfix mail system in 1997 and still maintains most the system himself. Venema talks to us about the unique structure of Postfix that gives it a leg up on security, “open” vs. “closed” code and thoughts for an advanced Scan project.

Q. Postfix touts security as a big feature because of its unique design. Tell us about the design and how the strength of Postfix’s code base has matured since you first wrote the code.

A. The distinguishing features of Postfix have always been easy configuration, high performance and the absence of security bugs. Postfix was designed as a collection of small programs loosely coupled together so that over time, adding functionality was mostly a matter of adding more small programs. While it takes more planning to implement a system as multiple small programs, maintaining small programs is easier than maintaining large ones.

Since 1997, Postfix has matured relatively well. It’s been used on everything from laptops to ISPs with tens of millions of user accounts. According to a survey of Internet mail server market share from 2007, Postfix ranked second after Sendmail, meaning a lot of people depend on the Postfix mail system.

Q. How does Postfix avoid security defects, which it’s known for, and what happens when the expectation of “zero defects” can’t be met?

A. From the beginning, absence of (security) defects has always been one of the primary goals with Postfix. Postfix avoids security defects through careful design and implementation. Defects are eliminated through testing and code review, and they are mitigated by layers of safety nets that are built into Postfix.

People have come to expect that Postfix will rarely, if ever, have a vulnerability that affects their systems. In the rare case that a vulnerability is discovered, it is handled by announcing updates for each supported Postfix version, in coordination with vendors who have Postfix in their software distributions and other organizations with a legitimate interest.

Q. What do you see as today’s biggest code quality issues?

A. The biggest concern is that software quality suffers as programs are modified to add features or to correct defects. I avoided this to a large extent by choosing an architecture that is based on many small programs.

With Postfix, many new features can be implemented by adding more small programs without modifying existing programs. Changes in one program do not affect the integrity of the other programs.

For the most part, Coverity’s “software integrity” concept on performance, security and quality is in agreement with Postfix’s primary goals. While easy configuration is not directly a code quality issue, I include it as a primary goal, because there will be fewer mistakes when a system is easy to configure.

Q. You work for IBM, but you also maintain an open source code base. Do you see an advantage to being proprietary or open at times, and are there instances where you could see proprietary software learning or adopting best practices on quality and security from open source?

A. I prefer to answer the question in terms of “open” versus “closed” source. There are commercial developers who directly contribute to open source. My employer is one of them. In the case of Postfix, many eye balls do not seem to make a difference for code quality. The few people who suggest code improvements are also the ones who make code contributions.

To answer the question in terms of “open” versus “closed” source, I know that some “closed” source developers are doing routine code review and automatic defect scanning. Both open and closed source software face similar pressures with respect to software integrity, to the extent that they are operating in the same market. For an example of this, take the closed source Internet Explorer web browser and the open source Firefox web browser.

Q. Over the past 12+ years, Postfix’s reputation has been based on having an unusually low defect rate. What prompted participation in the Scan Program and what would you like to see from the Scan Program that would add real benefit to your job as an open source code maintainer?

Postfix’s reputation has been strong for years because of its low defect rate, but the primary reason to participate in the Scan program was to reassure our quality control. Until now, Scan has found only five defects in Postfix: two defects in code that is not used, one defect in a safety check for a condition that never happens, and two defects in code paths that execute only when the operating system (not Postfix) is in a state of collapse. Scan reported 16 potential defects that turned out to be false alarms. I must admit that I would not have found these errors by testing or during source code reviews.

The defects found were not real problems with Postfix – they were defects in code that never executes or in tests for conditions that never happen. With projects that don’t do routine code reviews, I expect that Scan finds real problems.

Things that would make the Program more useful for Postfix: Scan could be made smarter about control flow with setjmp(); it incorrectly reported missing initializations in Postfix. It would also be nice if Scan could be aware of control flow through different code blocks that are guarded by identical tests; this would eliminate some of the false alarms. Symbolic execution would be cool, too.

Linux Code Security, Today’s #1 Code Quality Issue and What’s Next for Software Development

June 30th, 2009 Post a comment »

Don Marti is chair of Open Source World (previously LinuxWorld). A Linux user since 1994, Don has been a writer, editor, professional services consultant, and conference organizer. Marti walks us through Linux security from the developer perspective and shares his thoughts on today’s biggest code quality issues, open source advantages and best practices for proprietary and OSS teams.

Q. So you’ve been coding with Linux since practically the dawn of the kernel. What is the appeal for you? Is Linux an obvious choice for certain developers and, if so, for what reasons?

A. The advantage of Linux is that you can keep your suppliers honest by being able to go elsewhere. Senescent IT companies either fail, or squeeze their users for short-term revenue at the expense of quality, or both. Linux lets the actual product survive a company’s end. The other advantage is that you can strip out the functionality you don’t need. Your security and quality risks are lower when you can turn stuff off, either at the source code level or the install level.

Q. In the time that you’ve been involved with Linux, have you observed if the “strength” of the code base grown or changed?

A. I’ve been running Linux since 1994, and in that time almost everything has been rewritten a couple of times. The complexity has grown a lot, but the level of work you can get done with an out-of-the-box install has grown way more, so I can’t complain.

Q. From your experience, what has been the Linux community’s perception of quality and security in the Linux code base? Has concern been voiced and addressed and how?

A. There has always been tension between fast feature changes and code quality. Everybody has a different comfort level, and almost all users want a version that’s stable and tested and never changes — except, of course, for the one essential feature _I_ want.

Q. Has concern for Linux code security been tackled from a community manager perspective?

A. For Linux proper, the kernel, no. There’s no “community manager” for the kernel developers, and I wouldn’t wish that job on anybody.

If you expand the question to include all the software subsystems that go into a “Linux” install, then yes. Many of them have community managers and well-established security protocols.

The active distributions are doing a good job on security advisories and software update services, but users who choose to run software outside a distribution’s package manager are still going to be playing the same security game that users of other OSs have to. In the long run, as more and more software gets delivered through the distribution, the administrator’s job gets easier, but running a full-service distribution gets harder.

Q. As you know, Coverity is a big advocate of software integrity, by which we mean addressing performance, quality and security as a holistic effort. That philosophy underlies the Scan Project. What do you think about these concerns? Does one concern trump the other?

A. If you don’t have functionality, including performance, security doesn’t matter because nobody wants the software. If you don’t have quality, you don’t know if you have security or not. The difference between just a bug and a security bug is somebody coming up with a way to exploit it.

Being able to test and deploy a bug-fixed version is critical.

Besides performance, quality, and security, another factor to look at is usability. If a user can easily turn off some unused functionality in an application, that could cut risks. Usability in administration tools is a security feature — make administration hard, and time-pressed administrators will just open stuff up until the errors stop coming.

(A lot of malware spreads because the instructions for spreading it are less confusing than many of the instructions for using legit software. The secure way to do something and the easy way to do it need to overlap as much as possible.)

Q. As a developer, what do you see as the biggest code quality issues/concerns?

A. Web applications. It’s much, much easier to turn your static HTML site into an application — a fun discussion board, game, mashup, or whatever — than it is to learn about cross-site scripting attacks, SQL injection, and all the risks of running a web service. The web opened up development to lots of people, but you can’t do a new developer’s security learning for him or her.

It’s going to be a challenge for the authors and packagers of infrastructure software to give all those creative but inexperienced web developers a safer environment. Virtualization and SELinux work are both important here, but again the challenge is making the administration straightforward enough that people really use it.

Q. In your opinion, is there an open source advantage when it comes to maintaining code integrity?

A. Yes, for end users who do their homework. If you just download an open source application, deploy it, and forget it, there’s no real security advantage.

If you have software installed, you need a source of trusted updates. You could have an individual download that scored very high on any integrity metric you want at the time you downloaded it, but if there’s one exploitable flaw discovered later, it’s a problem for you.

You can’t just look at integrity at one point in time — you have to, first, be able to practice “software avoidance” by configuring your system to remove or disable unused software, and then, for the software you do need, you have to think about the whole process by which you would become aware of a problem and fix it.

Q. What could proprietary software development learn from open source development?

A. Open source projects are often more willing than proprietary ones to throw out a homegrown component for a better tested externally developed one — a good practice commonly understood by those who contribute to open source communities.

The main advantage open source has is culture and training. Proprietary vendors should encourage employees to participate in open source projects that don’t conflict with their work NDAs. (Having employees in the open source scene is also a good recruiting tool.)

And, some proprietary companies are catching on — look at what Apple is doing with SQLite as a storage back-end for proprietary software and see if it makes sense for you.

Q. What best practices do you feel the OSS community and proprietary software developers should adopt from each other with respect to code integrity?

A. Everybody should be using a distributed revision control system with strong hashes for all changes back to the beginning of the project, and everyone who’s allowed to see the code should have a full tree checked out. There’s no excuse for “one server gets compromised, we don’t know what got changed.”

Anyone on the project should be able to do a full integrity check, build, and test quickly.

Q. Any final thoughts on what’s next for software development?

A. The biggest change, starting in the mid/late 1990s, was open source companies. So now that there’s so much “commercial open source” you can’t really use “commercial” as the opposite of “open source” any more. And many vendors with an essentially proprietary business model will release the non-paying stuff as open source.

Very few users are going to be all one or all the other, so there won’t be much value in being the best in one category if there’s an excellent choice on the other side.

Everyone is going to be using outside-developed components as much as possible, with the level of “possible” defined by how well you can fit test-driven development and distributed revision control into your processes.

Ready for the 2009 SCAN Results? So are we!
Get them at OSCON in July. Until then, See What Open Source Leaders Have to Say on the State of Security in Open Source Code

June 23rd, 2009 Post a comment »

Having scanned more than 2 billion lines of code for over 600 customers and 265 open source projects, Coverity is the go-to expert in code quality and security for both proprietary and open source software.

Three years ago, we established the Scan Project with the U.S. Department of Homeland Security to measure the baseline for software security and quality standards in open source software.

Since then, we’ve been stress testing some of open source’s most popular projects with Prevent, our static analysis product that grew out of Stanford’s MC Checker program. Along the way, we’ve uncovered defects that range from simple superfluous code, to frustratingly subtle bugs, to exploitable security flaws.

In preparation for the next rev of our Scan Open Source Report, we’re kicking off some discussions with open source leaders about software testing, code analysis, how they’re tackling security measures in open source software and what proprietary vendors and open source projects can learn from each other when it comes to strengthening code.

At OSCON in July, I will be talking about the results of the 2009 Scan Report. If you’re going to be there, drop me a note or come see my talk. Until then, I hope you will find these conversations with members of the Scan and open source community useful in your own work to create secure code.

Stay tuned next week when we talk shop with Don Marti, Chair of Open Source World and long-time champion of Linux.

David Maxwell
Scan Project Director

News and Events