[Part I is here]
Java is popular because of one rather simple reason:
It's easy to use.
Java was touted as being a "simpler C++" when it was released: No pointers, no memory management (for the most part)...basically nothing that could over complicate the language, or make developers lives a living hell. This helped both existing C and C++ developers get up to speed with the language (due to similar syntax) and also allowed lesser experienced developers work out solutions with minimal fuss, and not have to concern themselves with complicated concepts.
Now, whether Java's simpler syntax was a good idea or not isn't what I want to get into. There are quite obviously trade offs to having a simple syntax (the big one being that solutions tend to get complicated as they grow...as some have already discovered). And at this point in time we're all well aware of the various strengths and weaknesses of the language.
The bigger issue I see is that the popularity of the Java language that came about from its ease of use now plays a major factor in how the language can, and should, evolve.
Having Java, or any language, gain popularity means that with the increasing interest in solving problems with it comes more basic grunt work getting done for people ahead of time. More developers solving the same problems results in patterns emerging from those solutions. Those patterns become libraries that other developers can leverage to solve even bigger problems.
As more libraries get created, other interesting problems get tackled. Frameworks emerge that allow developers to quickly work up solutions in entire problem domains, and not just some specific component of it. The frameworks evolve to work together, allowing for solutions that are almost trivially easy to put together. Development time drops considerably, as much of the boilerplate that would normally have to be constantly rewritten has been taken care of ahead of time.
Also, with such a large pool of developers to draw from, it's also very easy for companies to find developers that can hit the ground running with Java.
All of this makes good sense for businesses, as time-to-market is important. Currently, we've started to reach the limitations of what Java can do for us, and many have moved to other languages like Ruby and Python to further reduce that time to get a product out. And while those languages aren't nearly as popular currently, they may very well take that top spot from Java.
But in the meantime, Java tends to be the smart business choice as much of that grunt work has already been taken care of in the Java world.
Suffice it to say that I think popularity is a "good thing" overall for any language.
The problem I see at this point is that since Java is so popular it can not really move forward without making things worse. Unless Sun, or whomever starts maintaining the Java language once it goes Open Source, are willing to take drastic measures, they're going to continue to suffer from I would call Windows Syndrome...
Everyone knows Microsoft's Windows OS. Company shenanigans aside, it's become the single most dominant operating system on the planet. It's not the best out there, but it gets the job done for most people.
For years now, Microsoft has been struggling to move the operating system forward, adding new features and improvement wherever it can. In many cases they've succeeded, but they've stumbled quite a few times as well. And some have pointed out that a big reason for the stumbling is because of Microsoft's need to maintain backward compatibility with older programs and systems.
Microsoft's OSes have become so popular, and the market so large that they can't ignore situations where people are using older version of both their OS and applications. This ends up putting limits on how much they can change each subsequent version, since as long as people need those older programs to run (and sometimes it's a large number of people) they have to hold back the OS to support it in some way.
Basically, they're own popularity is working against their need to evolve.
Meanwhile, you have a company like Apple, whose systems are not nearly as popular as Windows-based systems (based on market share), which took a drastic step and broke backward compatibility with their older operating system when they introduced OS X, which was a major evolutionary leap from the previous versions. Arguably this has allowed the company to gain both market and mind share, and position their systems as being far more reliable and well designed than their Windows counterparts.
A big reason Apple could do this is because they don't have nearly the same size market as Windows -- they could afford to take the risk.
Now back to language-land...
Sun has made a conscious decision both on past language enhancements (Generics) and it seems on future ones (Closures) to maintain backward compatibility with both existing code and JVMs. This puts it in the ugly position of having to ensure that any change made to the language works within the framework of what's already being used by developers.
A big reason for wanting to maintain this backward compatibility is because of the large market for Java development: Breaking backward compatibility means they risk alienating a significant chunk of the community.
Meanwhile, you have other languages with a smaller market share/less popularity (Python and Ruby, for example) that are able to evolve and make changes quickly due to the fact that any changes they make aren't going to have as significant of an impact on its user base.
Hmm...this sounds familiar, doesn't it?
Maintaining backward compatibility for Java is a worthy goal to reach for, but for some of the changes that get considered, you can't necessarily take that option off the table. At the same time, Sun can't afford to break backward compatibility if the language is to continue to be popular, so we end up with some non-optimal solution added to the language that potentially makes things worse for the developers. It's a no-win situation.
This is where other, less popular languages have an advantage: Python in particular has broken backward compatibility several times in previous versions of the language. One big reason it could do this was because the market for Python developers is smaller than it is for Java.1 If Python were in Java's position, I doubt such decisions to break compatibility would be so easy.
Much like Windows, Java's extreme popularity prevents it from evolving in necessary ways.
With all of these changes that are trying to be made to the language, there is also the potential risk of decreasing the adoption of the language. Java's already widely in use throughout the industry, but if we expect new developers coming into the programming field to adopt the language it has to continue to be both useful and approachable. Adding complexity (as I think Generics and Closures do) works against this very goal, as it makes it harder for newer developers to get up and running with the language, and understand existing code.
(Of course, the ever-growing Java API also has this effect on new developers as well. Trying to navigate the disgustingly huge number of APIs in the Java SE is a daunting task even for seasoned developers...)
There's already evidence that Java is failing at this approachability, as there are colleges teaching Java that don't even try to talk about Generics when presenting the language to their students.2 If this is the case, then how do we expect these developers to work with features like Closures?
Another facet of this is that, with the added complexity we will undoubtedly add to the language when tacking on new functionality (either through trying to maintain backward compatibility or otherwise), we're forgetting a whole swath of current Java developers who, whether we like it or not, are going to severely impact the overall productivity of the community. And making the language more complex is just going to exacerbate an already bad problem these developers represent.
One thing I've grown to understand while being involved with Java-related projects is why there's so much hate towards the Java language and perhaps the community itself: The bulk of it comes down to the overall makeup of the Java "community". I use quotes here because the community I'm referring to isn't just the vast number of intelligent, enthusiastic people that inhabit places like JavaLobby, or DZone, or TheServerSide.com, but the entire group of developers out there that have made a career using the Java language.
The unfortunate consequence of having a language that's so easy to use is that, while there's plenty of very capable and smart software developers using the language, there is a vastly larger group of people that have been using Java out there who are...shall we say...less than enthusiastic about the whole art and science of software development. You know who I'm referring to: Those unfortunate (and aggravating) souls that have no desire to really learn anything more about programming than what is necessary to earn them a paycheck - the ones from whom we regularly get our laughs on sites like The Daily WTF.
Let's face it: There are a lot bad programmers in the Java world.
It's the result of the combination of the language being both approachable, and the community so large (since the more people using the language, the more likely the chances of lesser-skilled developers using it). There are quite a few of these programmers using Java, and they are not going to go away as long as having "Java" on the resume is a guaranteed chance at snagging an interview, and possibly a job. In my career as a developer working on Java-based projects, I don't think I've ever had a job where there wasn't at least one of these guys involved in a project I was on.
Now, granted this isn't necessarily a Java problem so much as an industry problem. As someone's recently pointed out rather astutely, the software development world seems to have a interesting characteristic of allowing those of both lesser capability and ambition to coast along doing programming work. So it stands to reason that since Java is so popular, the bulk of these lackadaisical programmers end up working with Java as opposed to some other language.
Unfortunately for Java, this ends up painting the language in a bad light, as there are so many of these developers using the language. The fact is that you could substitute some other language as the most popular and probably have the same result. Yes, the Java language itself does deserve some of the ridicule it receives, but I think the bulk of the ridicule is more a case of disliking this specific segment of it's user base.
At any rate, this is the group that comes to mind whenever I read about adding something like Closures to the language. It is this crop of developers that has the potential to do the most harm when new features get added to the language. Especially when those features add additional complexity.
Because what happens is that any code these non-enthusiastic, non-creative, unthinking lackey programmers create is likely going to be some bastardization of what was originally intended as the use for said features. They will drop generic classes (and possibly even closures) into their code like it's nobody's business...and turn what could have been a simple project into a nightmare.
They will basically use said features just for the sake of using them...without any clear understanding of why or how they really should be used. And they won't care either, because to them it's just a job.
And imagine that situation repeated over thousands of projects.
Of course, I realize that "because it'll be abused by bad programmers" isn't the best of arguments against adding new language features to Java. For one thing, there are ways to mitigate this sort of problem: Code reviews, good hiring practices, etc. But considering the size of the Java developer base, the odds of weeding all of these kinds of programmers out is about the same as the chances of wiping every cockroach off the face of the earth...about nil.
While the best of us out there will be able to work very intelligently with any new additions to Java, these types of bad programmers appear to be so prevalent in the Java world that it can have a very real impact on productivity for everyone - including the smart ones - since somebody will ultimately have to deal with the code that these developers produce at some point.
Eventually, the misuse of these new features (good or bad) could end up working against the Java community as a whole - resulting in Java gaining a bad reputation for creating solutions that are unnecessarily complex.3 And the reason for this may only be because developers - the bad ones - didn't use the features the way they were intended and made things much worse than had they not had the functionality available at all.
Sometimes the only solution to making sure that people don't shoot themselves in the foot is to make sure they don't have the gun in the first place.
One Step Forward...
Overall, I'm not opposed to improving Java, even if those of lesser skill can't quite get their heads wrapped around the concepts that get included in the language. And I'd rather see some improvements over none at all.
But I'd much rather have these improvements to Java done right. And in some cases, I think that means breaking backward compatibility. It's not an easy decision to make, but I think those in charge of Java's stewardship should consider the option. We all want to see Java continue to be successful, but if we insist on trying to contort additions to the language to fit the existing system we're going to end up making things harder on developers, not easier.
And we have to be cognizant of the fact that, despite our best efforts to make programmers' lives easier, we're potentially giving them something that they won't fully understand how to use properly and may make things much worse overall for the Java community.
(Stay tuned for Part III)
1 - I am not trying to make a "Java is better than Python" argument here, so save the ranting and raving. It's just a simple fact that Java is more popular than Python. That doesn't make Java better than Python. And I would certainly not try to make that argument...as I actually believe quite the opposite.
2 - Of course, the subject of teaching Computer Science in colleges is just as complicated as this whole discussion over Java...and I don't want to diverge too far from the point here. Let's just say that the fact that Generics aren't taught is both because of the complexity it introduces and because students aren't necessarily taught everything they should be, and leave it at that for now.
3 - Yes, I know, people could argue that Java already has a reputation for resulting in complex solutions...but doesn't that make the situation just that much important? If the Java is already considered to make solutions that are too complex, then what's going to happen when we add further complexity to the language?