Harmonia and Music Notation Editing

By: David Psenicka

I'm a composer and one of the developers of Harmonia. For the past fifteen years I've taught classes in music theory and technology and have given private music lessons. One of the most tedious aspects of it is having to notate scores, often on the fly and requiring a lot of detail. Many of the decisions involved in score editing (especially for assignments) are repetitive, and popular score editing programs like Sibelius or Finale require a good deal of time and patience to coerce the score into an acceptable state. I realized this was a common issue for other composers so I wrote an open source application called FOMUS that simplifies the process of notating music by automating many of the mechanical decisions involved. Although the software was written for algorithmic composition, the work I did on it altered my approach to composing and teaching as I was able to reduce a lot of the repetitiveness involved and devote my time and energy to more important tasks. It was also a fascinating project as it involved solving many difficult technical problems that arise from modeling a complex system of symbols and concepts in computer code.

In my experience teaching I've always felt the need to customize my teaching materials for my students and felt that there should be a more efficient way of doing this. This is what motivates much of the work I'm doing on Harmonia. I'm currently developing many of the editing capabilities of Harmonia, which includes the notation engine and text editor. The interface along with the notation generators is designed to be easy and intuitive to use so that an assignment or test can be put together quickly. Less time and effort spent on creating custom materials means content better tailored to students and more time devoted to higher aspects of teaching. From my own experiences as a user of the software I'm helping to develop I'm certain it will change the way teachers approach their craft and will improve the quality of music education for both educators and students.

What’s the value of Harmonia?

by: Rick Taube

“What’s the value of Harmonia?” Or, quite literally: what commitments and investments did we have to secure along the way to move an initial idea about how to improve teach music theory using technology into a functioning company? The first thing I should say is that it was a very long path (almost 20 years!) and much of journey was not easy. The first two years I worked completely on my own developing the analysis algorithms in a language called Lisp that I had learned at Stanford when I worked at CCRMA. It was a complete leap of faith -- i was putting in lots of hours trying to get a program to analyze harmony quickly and accurately, and to "communicate" its results back to a user. The time I spent doing this work meant I was composing less and working less on my public-domain composition software, Common Music. I don't think I even listed this research work on my faculty reports because I had no concrete results to talk about yet. But once I had a functioning analysis program, I quickly learned how hard it is to convince people about a new idea unless it is something they can actually visualize. I showed my program to an academic dean, and while he was nice about it, he didn't really understand the issue or what he could do to help me. He sent me to a music software publisher he knew and the same thing happened, only that he understand my idea even less than the dean. I talked with the Grants and Licensing office at my university and they decided the app wasn't worth anything and gave me complete rights to the software. Realizing some of the hurdles I would have to overcome, I continued to work on the software, adding a PDF back-end so the programs markup became visible and the program's explanatory text was colorized according to they type or severity of the analytical issue. Seeing the results in a semi-intuitive way had a real positive effect on viewers, and I decided to write an article about it for the Computer Music Journal. CMJ is a top-tier publication in my area and when they accepted the article it gave me some credibility. The article gave an clear overview of the software and also included images (see below), which also helped people understand what it was doing.
The 1999 article predicts the Harmonia cloud, realized 13 years later.

My first nibble for actual support came from McGraw Hill. I showed it to a theory textbook representative and he "got it" instantly! He was also part of the on-line component for the company and understood the possible value it had in that new industry. I signed a contract, but in the mean time McGraw Hill was finalizing its sale to a larger publishing company. This new owner wasn't interested in on-line learning as much and all pending, unfinalized contracts, including mine, were cancelled. At that point I thought I was at the end of it, and I went back to writing music and also a textbook. I hadn't really given up, but I just wasn't sure what I could do to move the ball forward. The next real change happened a few years later when a graduate student, Andrew Burnson, an outstanding programmer with experience with music notation, entered our composition program here at UIUC. (UIUC has a long history in computer music and we've be fortunate to attract a number of composers/programmers like Andrew over the years. Other composers join our program and find out that their ability to think abstractly about music processes and systems and to notate these abstractions in symbols, often makes them "naturally" good programmers. I pitched an idea to completely rewrite the software in C++ app with multi-media features and Andrew was immediately interested in the idea. I was able to secure a $6,700 Creative Research Grant from FAA, and with Andrew on board we came up with the initial design and creation of an app called Chorale Composer. Meanwhile Rachel Mitchell starting teaching music theory at the UIUC and she got excited about the pedagogy that the app could potentially support, and agreed to start developing content for the app and using it in her classroom. Over the next few years we collectively applied and received several more grants -- two from PITA (Provost’s Initiative on Teaching Advancement) and one from the Campus Research Board (which won the Arnold O. Beckman Research Award for projects of special distinction) totalling about $26,000. Along the way we "retired" Chorale Composer, and then Halim Beere joined our team and we started working on Harmonia, which was a generalized version of Choral Composer that could send and received course data from a server. Meanwhile I began talking with the Office Of Technology (OTM) and they sent me over to EnterpriseWorks, a very successful startup incubator at my university. I made a pitch and they took us on at a 90% funding level. This was a pivotal moment for us because it allowed Halim and I join the iCorp (Illinois NSF Innovation Corps) which is basically a boot-camp for startups that provides very intense training and leads project leaders through the process of "customer discovery" and building a business canvas see if the idea is actually marketable. That same year we receive a Proof Of Concept Grant for Harmonia from UIUC's Office of Technology Management for a total of $15,607 to complete the work on the Harmonia prototype. All this momentum, plus the support by EnterpiseWorks, got us into the company formation phase, in which we developed a strategic plan, secured intellectual property protection (trademark and patent) and ultimately led to our application for an $225,000 NSF Phase I grant, which we received in 2015. That grant allowed us to hire David Psenika and Ming-Ching Chiu, and all of us worked together to complete the network prototype in Fall 2015, then test it in a real course (Music Theory I (MUS 101) at UIUC) for a semester. From that experiment we were able to verify the application worked and gave better results than the status quo for teaching music theory. The rest, as they say, is history!

How did the Harmonia music theory app start?

by: Rick Taube

The idea for Harmonia came to me in 1997, just two years after I joined the composition/theory faculty at the University of Illinois. At UIUC all composers teach theory, and as the newest faculty member I was assigned to teach first-semester theory and aural skills. While I had taught similar courses in the past, during the preceding decade I had actually been working outside academia, as a computer researcher, first at the Price-Waterhouse Technology Centre in Menlo Park, CA and then for five years at the Zentrum für Kunst und Medientechnologie in Karlsruhe, Germany. What struck me most about teaching theory again was that — though the content I was teaching had changed very little — the circumstances in which I was teaching was very different than anything I had previously experienced.

I firmly believe that the best way to learn music theory concepts is to study how they are used in real compositions and to compose music that demonstrates those concepts. To do this, it is important that a student receives fast instructional “feedback” from teachers to complete a learning cycle. I was taught harmony and voice leading by Prof. Herbert Nanney, organist at Stanford University, who lectured to his (small) class sitting at the piano, where he would play, improvise and work with us “in real time” to improve our exercises. My Music 101, in comparison, took place in a large classroom and had about 90 students with widely varying degrees of theory knowledge and skills. I had eager TAs, but their theory experiences were also quite varied. In my class, I found it was simply not possible to provide each individual student enough "learning cycles" to really master concepts and remediate problems. Since hiring more teachers or limiting student enrollment was not an option, I realized that the best way to address these issues was to adopt a technology that could provide unlimited access to practicing analysis and composition with instructional feedback, thus breaking the linkage between class sizes and the amount of learning cycles that can take place. Moreover, unlimited guided practice would help every student learn, regardless of class size. Since I have strong programming skills I began working on developing music software that could analyze real music and relate the issues it discovers back to a user in detail. Progress was steady but also slow as research grants for faculty in the fine arts were hard to come by and did not involve large sums of money. The game changer came for us in 2015, when the National Science Foundation awarded Illiac Software a $264,000 STTR "proof of concept" grant. With that grant we were able to complete a prototype, embed it in an actual course and prove that computer analytics can indeed be used to improve students learning outcomes over the status quo. It also facilitated a number of other beneficial features such as integrated multimedia and cloud-based course delivery! Our NSF grant took us through the end of 2016, and we started officially commercializing in January 2017. Our primary goal as a company is to improve theory education for all students at a fraction of the cost of what what would otherwise be possible. While we are really just at the start of this journey, I am excited and heartened by the feedback we have had from professors and students alike, and we have many plans and ideas to improve our application and course management system as we move forward.