School district builds own software to handle student data, defies doubters | Hacker News Reader

School district builds own software to handle student data, defies doubters

timescolonist.com

76 pointsjaphyr58 comments

I'm pretty familiar with this problem space having worked in both a state department of education and a school district.

I would not expect this to be a long term success. Virtually every district of reasonable size starts with a homegrown and maintained solution and ends up going with one of the big guys for lots of reasons.

1. Security/liability. You have to have pretty damn good systems here and definitely want to point at someone else's assurances and contract that allows for recourse.

2. Rapidly changing business requirements. This is everything from state and federal reporting requirements to the expectations of end users. It's hard enough to maintain these core systems and keep folks around who are good enough to do it. Now deal with changing technology (oh you want a parent portal? Oh, you don't want to use Flash/Java for iPads? Oh you want ...) and you really can't attract and retain the kind of software people you'd want to continually redesign, refactor, rebuild, extend, and expand. Development on these systems never stops, which is why it's often easier to just drop off the old and buy the new hotness. It's also why scale is so helpful with these systems. If your users want this, someone else already requested it or will shortly. A lot of the improvements in these systems happen because of how many users they have and can be made because the cost is spread across all those users.

3. Interoperability with other systems. Most of these systems are not omnibus catchall solutions for the entire business needs of a school district. That's probably for the better-- the folks who try to do that end up doing it very poorly. But you will want your HR and payroll to talk to your student information system that should probably talk to your assessment system that should talk to your curriculum management system that should talk to your special education IEP case load management system, etc. It's hard to do this and even harder when your solution is built to work in one context. Standards here have only barely helped. Middleware is expensive, shitty work. Again, not easy to attract killer talent and it is a constant need. The big boys have everything else built and just focus on middleware and have probably already worked with and written what's required for all of the popular systems you are using in these other areas.

This is just the smallest fraction of reasons it's a huge PITA to write and maintain these systems. That's not to say that it can't be done but it is to say that this is not a small, well-defined problem space with just a few distinct end-user types and functional requirements that can be built once and then monitored and tweaked over time. Not if you want to do the real stuff that makes collecting and maintaining all of this data worthwhile.

While all three of your points are excellent concerns, I think you may be giving off-the-shelf systems a little too much credit.

The SIS systems I know (US, K-12) do a mediocre job of updating quickly to new technologies/requirements (#2) and do a terrible job of interoperating with other systems through APIs (#3).

Doubleplus agree. Have been working with a state department on a student registration system for the past few years, and now everything's going to be placed under one of the 'big commercial' players in this space. No interop specs have been delivered/offered, and the integration team seems wholly unaware of the complexities of the requirements laid out by the state for the types of processes we have to comply with. And talk about 'not adapting'? We have processes in our system relating to HR and the commercial team flat out said "we don't want to deal with that", but also balked at offering us any sort of integration service points (REST/SOAP/anything) to keep doing what we need to do next year.

I've never met a software vendor in this space that has a freaking clue what they're getting into when they sign these contracts. They never think about interoperability despite signing on to the requirements, they assume too much when it comes to their customers defining requirements, and hate offering up the goods to work with other systems you have in place (but can usually be twisted to offer bare minimum integration).

Would you be willing to touch on what the reasons were behind your department deciding to switch to a "big commercial" player?

I've honestly no clue - from my perspective it was a statewide edict passed down last fall - there's probably more to it but I've not been filled in.

I actually agree with all of that (also US K12).

That's why I think you're better off with something you can sever ties with when it falls behind on #2 and use clear RFP and contract requirements + in-house dev talent to drive #3. The hardest to deal with is always going to be #3 because everyone always is trying to get you to use their whole architecture, even thought most of the time they suck at some or all of what they sell.

I think that #3 will also get a lot better with commercial software. The reality is that most of these guys have spent the last 5-10 years buying up competitors who built other systems their customers were interested in as an attempt to get all that business under one contract. Their efforts integrating software from acquisitions has been pathetic, at least in part because they haven't had a services-oriented architecture even though that's what really is called for. So I think this is a major area where we'll see changes in the next 5-10 years.

That being said, I think pretty much all off-the-shelf systems suck, but you're better off being able to hold someone else responsible for dev and sustainably meeting requirements and use in-house talent to hack around and problem solve so that users end up getting what they need fast.

You also want to chuck systems every few years and get something a lot better without having to pay the upfront costs of a rewrite like you would with a BYO-SIS.

And while I think this goes without saying, though these days I've seen districts screw this up, no matter how hosting is taken care of you need to own that data. You need nightly backups on servers you control and full access to your own information.

Thoughtful points all around.

I'd add that your suggestion for #2 (migrate from one system to another every handful of years) has some pretty painful costs associated (PD/training, re-implementing customizations, data migration costs). At least when you've taken care to own your data/backups, this kind of migration is possible.

5-10 years is far too long to wait for system interoperability. My company (Clever - getclever.com) has bolted a read-only API onto a dozen of the top SIS systems today, would love to hear your thoughts (I'm dan@).

another good point. Our district has gone through five, count them five, commercial student information systems because the company has either gone bankrupt, been bought out by a larger vendor or depreciated. We just get our data converted, staff trained and somewhat comfortable and then the rug gets pulled out from under us again and we have to start over.

Anyone who think that commercial vendors are there to support education is dreaming. Commercial vendors are there for two reasons only - earnings per share and profitability.

gregg

while it may be a point that they may have to rewrite it down the road, the advantage is they retain the logic so they don't have to start again from square one with a new vendor trying to get their system to work.

Also they could opensource it or sections to get help in hardening the code, create interoperability APIs, or anything they choose. Such an option is usually not a possibility with a private vendor.

Another issues is when you do have a significant change in law or policy getting the vendor to get those updates out in a timely/inexpensive manner may not be so easy.

Interoperability is overrated, what happens when HR decides they to get a new system? ...and that new system doesn't work with your current tracking system X but only with the much more expensive and over complex Y pro system?

I guess it comes down to is how important is it to have your hands on the data and process? I know some schools are really tiring of the "new system every five years or so" cycle that they go through (this can include, besides software, new hardware, training, reassigning staff responsibilities - all depending on the systems needs,) having control of costs and process might help them in the long run.

Adding some extra on to my previous reply.

"2. Rapidly changing business requirements. This is everything from state and federal reporting requirements to the expectations of end users. It's hard enough to maintain these core systems and keep folks around who are good enough to do it".

In mid 2011 our state passed some new laws as to how funding would be applied for virtual school enrollments across the state. We had a model in place, then code rolled out, within a few weeks - we had one bug that reared its head a few months later that was patched in a day and it's been fine since.

I'm not convinced the 'commercial player' that's taking everything over even understand the concept of our funding model, much less will be able to reliably integrate it with their existing systems, in the next 6 months. That's after handing over the code (with some, but certainly nowhere near complete, tests).

They may still pull it off, but I've yet to see them react quickly to 'changing business requirements'.

We will absolutely not be moving to a commercial vendor with this product. Right now we are creating a not-for-profit society which will be the vehicle to ensure this is maintained as a shared-service dedicated to support IT initiatives in our school districts. We are very tired of being tied to commercial vendors whose sole focus is earnings per share and profitability. In any case please have a look at our business plan and see what our focus is in doing this.

It is a model which could easily be adapted by other jurisdictions whether in Canada the US or anywhere else for that matter.

We are always open for questions, suggestions or comments.

cheers

gregg

I've actually thought of proposing such a project for our school district. I'm not sure what we pay our current software, bu my thought was to develop a summer STEM program and give HS students the opportunity to participate is a real project with real consequences, etc.

Many of the school people (administrators/teachers) were very unhappy with the previous system, so I expected it to be a reasonably easy sale. Unfortunately I didn't have the necessary free time to pursue it much further than "thought experiment." Since, the school district has purchased a new system that school officials actually like.

That said, our school district (like most in Illinois) is facing some serious budget crunch due to the state failing to make payment on time (or at all), so maybe it's time to dial up this project again, and pitch it to the school board.

One of the big obstacles I see with this kind of model is the need to get security rock-solid. Any system that deals with student information is subject to federal privacy laws, and the consequences of a data leak would be pretty significant.

Any thoughts on how to deal with this? I am wondering if os projects like the one you describe could partner with an experienced mentor, but I still wonder what kind of structure you'd want to use to address liability issues.

Mandatory ACLs, built right into the database and integrated with the OS that the system runs on:

https://code.google.com/p/sepgsql/

Assuming, of course, that you have some good way to connect a user's login to the app (which is probably being delivered on the web in this day and age) with an SELinux role. Luckily, we can do that too:

https://bugzilla.redhat.com/show_bug.cgi?id=501228

While privacy issues are important, state compliance & reporting are actually the bigger challenge.

These types of systems typically track enrollments and attendance, which get reported to the state and determine the district's funding. The calculations here can be fairly wonky (i.e. some states require schools to calculate every student's attendance down to the minute) and an error could have massive consequences.

My suggestion for a project is rather than trying to replace the core database, use an API like Clever (https://getclever.com/developers/docs) to build something cool on top of the existing database. (Disclaimer - I'm a co-founder of Clever).

I work on a medical information system. You are grossly overestimating the ability of enterprise software companies to provide good security.

See below. The accounting problem is a far greater challenge that security. Interestingly school information systems are notoriously bad at security, so hitting par won't actually be that hard. Accurate accounting may be a huge challenge, though.

Either way, mentoring would be key, and no system developed vy students should be allowed into production at a scholl without a complete review. But that that's part of the process. Even in a private company/OSS project, code shouldn't be allowed into production without a review. I do have access to many people with the right skill set to be able to develop such a system on their own (my self included) in addition to the fact that I work for an information security research institution, so I have access to folks who know a whole lot more about trusted systems than I.

Of course, HS students will make mistakes, but the mentors will be there to review the work, and make suggestions, while keeping "bad code" from hitting a production system.

In the end, I'm not sure that I'm convinced that a student developed system is any different from one developed by some outside company.

assuming that the majority of the work at commercial company X is done by people who weren't long-ago HS students themselves, often with minimal training and support at their company, I agree, there's likely very little difference, other than the mistakes will be more open and easier for people to see and fix.

It is totally possible for school districts to develop their own software. I helped launch a similar project to create a mobile app for my school district when I was in high school. Quotes from ed-tech companies were much more expensive than our costs. We also made it open source so that other school districts could use it.

Edit: Links

[1] App (now also on Google Play): https://itunes.apple.com/us/app/ifusd/id454673943?mt=8

[2] GitHub: https://github.com/FremontUnified/

link?

I've done similar when working for a non-profit.

Having the tech guys on campus helps drive the development you get access to the people who work with the data as well as updates on requirements of what is needed (there's always something people will forget in a spec... until some report is due and they mention it.) They can also bring in staff to test components as they are being developed.

My thing was child care related; student tracking is a whole new level, key things are relationships as in families, adults to children children to education, teachers to children to classes to teachers to credentials, etc. Plus whatever factors affect all that (social assistance qualifications, immunizations, gpas, etc.)

Though once you get the core data figured out and organized (which is the major bit) the rest starts falling into place.

It's not rocket science but it is a definitely a detailed data situation, which you need people dedicated to it for years to put it all together.

As far as anything like that being "done" not in the public sector... you got everything going then some new law changes and you need to adjust tracking/claims based on some new factor or add in an adjusted reimbursement calculation etc. so you have to restructure some data add some new entry methods, new reports, etc. Easier on an open system than a closed system.

Problem with "canned" software is it is usually developed out of one district with their own unique needs when it gets packaged as a one thing fits all it needs to be patched for other conditions and environments... It is usually geared for a specific administration staffing method, so buying something canned may mean you have to do administrative changes in order to work with the new system, etc.

It's good work though, and a nice challenge to sort out all the data and get it collected, properly managed, and accessible to the right people.

For me, while it's absolutely no surprise, it's still quite depressing the clear lobbying that is/will occur to trample this project. As others have said, the cost seems quite realistic and I would put my money on the 8-person having a much better handle on the end-users needs over some big consulting-ware shop. But the big firms surely have great influence with those most able to appropriate the funds for whatever solution B.C. opts to implement. There's obviously maintenance, support, enhancement, etc. that must go on after initial launch, but that still should not get anywhere near the monster price from an outside vendor.

I hope their fate is not so dire. I will definitely be interested in updates on the school district's efforts.

It's a little amazing, when you think about it, that a lot of people think you need to spend $100 million to build software to manage grades and attendance.

Especially as geeks in the startup world often build far more complicated software on budgets quite a bit smaller than $5 million.

Even if all it did was grade and attendance. It is not trivial to get that right. My own little pet project http://gradezilla.com took years of refinement before it could handle just those things. Tying in with other aspects of a schools system makes it many multiple more complicated.

That's fair, but it's still not a $100 million investment.

$97 million for this http://en.wikipedia.org/wiki/BCeSIS. The BC government was milked. A system for a few thousand school administrators costing that much and could not handle the load? The ineptitude boggles my mind.

Awesome to see there are some smart people out there fighting back.

In the late 90s the San Francisco school district was sold a PeopleWare system, whereupon they paid consultants millions of dollars to customize the package. 'Twas a failure; they tossed it all away.

I don't remember the details, though I was feeling their pain, having to deal with (a) the unbelievably horrid, steaming pile that is PeopleWare, and (b) the clouds of consultants that start calling you up and pumping you for money once it is out that you're doing an install of the pile. (Hint: You don't need any help from a high-price consultant to install PeopleWare. You need a can of gasoline and a match).

So the thought of a school district doing this on their own makes me smile. They could do worse!

Technology and software in schools in one of the most corrupt, backwards, and destructive markets around. Schools must take charge of their own software and destiny with a robust open-source economy.

The Science Leadership Academy, a public high school in Philadelphia, runs on an open source SIS we've developed for under $100K and it has been in production for over 3 years. We are working now build an alliance between several like schools to share the maintenence and development costs of supporting a public release.

If this group is planning on spending $5 million, we're still dealing with enterprise salesman hiring amateur or out-of-date developers. Give me another $100K and I'll put money in hackers' pockets to build quality software and make sure no other schools _have_ to pay for the same thing twice. This is more of the same crap, there's a whole new generation of developers that don't have access to the school software market, and it's students that are losing out because of it. The developers that built all the sites and apps this community knows and loves wouldn't need anywhere near $5 million to build this SIS and it wouldn't look like it got left over from 1999

1.5 million, close to 2 years and a "small band" of programmers to deliver a specialized product working on-site with the end users. Call me naive, but this sounds completely reasonable to me.

I think the "doubters" are just not used to seeing that sort of thing. They probably think the approach is going to look more like this:

http://topics.nytimes.com/top/reference/timestopics/organiza...

You know, the case where you hire a team of contractors to build something, and then that team totally screws up, goes years past the deadline, and charges you enormous amounts of money. Meanwhile, there are ready-to-ship products out there that can be bought for a yearly fee, which carry less risk but higher costs.

Agreed. I'd go further and say I would expect many here have successfully completed similarly complex projects single-handedly with a lot less time and money.

As I grow older, I'm increasingly starting to believe that many of the advancements in sophistication of software are actually somehow making it much less possible to successfully implement relatively complex solutions on budget - I am making this comparison to the relatively short era where platforms like PowerBuilder/Delphi/MSAccess were popular. If you mention on of those products now for literally anything, you'll be met with howls of laughter, but for the life of me I don't understand. Meanwhile, spending $2 million on a project that is laughably short on functionality and seems to take weeks to make the smallest of changes is considered (with grave seriousness and 100% infallible confidence) to be perfectly proper.

I think the keys are that they have a limited set of requirements, they're close to the users, they have low/reasonable expectations, and are willing to accept that the system will need updating & improving in the future. Sounds like a perfect fit for one of the agile methodologies.

We are currently on budget and on track with our build. Incidentally when it comes to attempt to build an open source package to replace a commercial product this is not entirely correct.

Before we began our build we did a complete analysis of the functionality we required in a SIS using our current commercial SIS as the basis. We found that because the commercial SIS was developed for multiple jurisdictions the actual screens and fields we required for British Columbia schools was only 27% of the complete commercial SIS. So we are not crafting a replacement for the commercial but a rather small subset of the requirements targeted specifically for our environment.

hope that helps a bit.

I am watching this project pretty closely, because I believe pretty strongly that there is much room in education for some high-quality os software. I also believe this software has the potential to facilitate some pretty significant change in public education. I really want to see some high-quality os solutions become prominent in the education world.

The software development world has had numerous os tools for decades: vim, emacs, apache, to name just a few. I can't think of anything equivalent in education. Most teachers are still writing their lesson plans in Word or its equivalent, which is like writing your code in notepad without any coding-specific extensions.

There is lots of room to improve ed tech, and some of these improvements need to be open source to have any meaningful widespread impact.

"Most teachers are still writing their lesson plans in Word or its equivalent, which is like writing your code in notepad without any coding-specific extensions."

There are plug-ins for Word that can produce quizzes, and most of the interactive white-board software packages (each dedicated to a make of interactive white-board) can import from Powerpoint or allow authoring of interactive content. What I as a teacher don't need is someone telling me to use some huge IDE like package to build my lessons!

In the future I'd like to use a cheap tablet to scribe on and have what I'm scribing (or what I have prepared earlier) just appear on the projector and on the student tablets. I'd also like to be able to invite student A to 'show' something to everyone else. Then I'd like to be able to roll the whole lesson up and pop it somewhere like a Moodle course so the students can access the stuff later for revision.

The original article addresses 'back end' student tracking and data capture type software. More of a business application.

I was talking about a different project that I would like to see happen, but built using a similar development model. To be more specific, I'd like to see a dedicated curriculum development tool built. I;d like it to be high-quality and open source, so it can see widespread use in schools of every economic level.

I am not envisioning a simple plugin to a word processing tool; I am imagining a dedicated curriculum development tool that manages things like importing standards, and connecting detailed parts of a learning plan to bigger-picture summaries.

I in no way imagine such a tool being mandated for anyone; I simply imagine a tool that is powerful enough that people will see it used well by experienced teachers, and want to use it themselves.

"I simply imagine a tool that is powerful enough that people will see it used well by experienced teachers, and want to use it themselves."

Well, good luck, and if you have thought through the detail of this I would be interested to see the specification (in the Spolsky sense) for the system you envisage.

"I in no way imagine such a tool being mandated for anyone"

Of course everyone will have to use it!

Edit: parent poster is also a teacher, so he must know that anything that is suggested by management as a central tool will immediately become mandatory!

I am hoping to complete my MEd in December, and I want to complete an mvp of an os ed tool as my masters project. I wrote a bit about the tool I envision previously [1], and I recognize that the next step is writing a proper spec. That will be part of the masters work.

Nice to see another teacher on here!

[1] http://peak5390.wordpress.com/2012/02/07/why-do-programmers-...

Good luck with the more detailed specification. I shall read it with interest. In the UK we tend to have many small outcomes per lesson, and a second document called a scheme of work tracks the large arc of a year's work.

Hey man, I feel like your interests and mine intersect? Do you have any personal Twitter/microblogging streams I can subscribe to? :)

I have some contact information in my profile, and I'd be happy to hear from you or anyone else.

Thanks for this japhyr. Our project is proceeding on track and on budget and we are going to be piloting in elementary schools by April of this year. Our primary focus now is reporting and business intelligence which will not only produce embedded reports but allow for Ad Hoc and true BI capability so schools and teachers can get real analysis capabilities in a user-friendly dashboard.

gregg

" By taking a not-for-profit approach and using freely available open-source tools, Saanich officials expect to develop openStudent for under $5 million, with yearly maintenance pegged at less than $1 million."

They estimate that using their old software would cost $100mm over ten years. They're reducing that to roughly $15-20mm.

Government software is lovingly expensive. The state I live in spent over $30mm to rebuild a purchasing application. And it doesn't even work!

Go figure.

I am the Director of IT for the Saanich school district which has incubated this project. There is a lot to say about how we got here but this is a somewhat different project than most other open source student information systems (and all commercial products that we are aware of).

Firstly it is build from the ground up to work as an enterprise system. Meaning rather than just function for a district and its schools our system is architected to function at the provincial or state / district / school level.

The system is designed to accommodate cross-enrolment between schools within a district and between districts. As well it can accommodate continuous entry (year round schools) and is built for conventional as well as distance learning schools.

It is also has a sophisticated security model which considers users at the enterprise level, district level, school level or any combination thereof. In other words you can be a vice-principal at one school (with that view), a teacher in the same school (with that view) and a parent with children in another school district.

The system is also built to work in a distributed hosting model where the primary can be at one site and secondarys at other sites. With the next version of PostgresQL it will be possible to have multiple primaries.

There is so much that is different about this system from the standard SIS. With its capacity to function at the enterprise system it could accommodate an entire state or province.

Right now the system has been licensed under the Education Community Source license (modified Apache 2) to ensure that we have better control of the code but we are definitely looking for input and help into the system and it will be available to other jurisdictions when it is complete. Have a look at the documenation on our website and I will try and answer any questions or comments relating the the system.

Thanks for all the supportive comments.

Gregg Ferrie

PS if you wonder why this has gotten this much support in a relatively small school district we had already converted most of our servers and workstations to Linux-based diskless clients (approximately 2500 in 18 schools). All of teachers and students are used to using open source software including LibreOffice, Scribus, The Gimp and many more.

Oh man, how awesome would it be if a high school CS class were able to have a lesson where they learn how to contribute to an open source project that their own school uses? So cool!

I hate to be Donald Downer but ... it's not done, yet.

' close to finishing the core version '

'expects to begin testing its openStudent software in elementary schools this spring, with a full launch to follow in 2014. An advanced version for middle and high schools is slated to roll out in 2015.'

True but I have a lot more faith in a tightly integrated 8 person team than some off the shelf bloat-ware customization project.

As Do I, it sounds like they are partially doing this to prove a point, which means they'll likely find a way to go through with it.

As would I.

I don't assume they _are_ tightly integrated - all the article claims is 'eight people'.

These could be crack coders. Or they could be '1 guy who kinda knows VB and 7 people learning OJT'.

Just to clarify our staff consists of myself (IT Director for the Saanich SD) - I am the project lead/sponsor for the district. The eight people include: project manager, product tester/quality assurance, stakeholder business analyst and 5 strong Java programmers, including one developer who is one of the original creators of the PostgresQL database.

gregg

Fascinating - thanks for the update.

I wonder if they considered throwing their lot in with Schooltool:

http://www.schooltool.org

It has been around a long while now, is in use around the world, and should at least make a good starting point for further customization.

I can't even load the BCeSIS website through all of the Java security warnings: http://www.bcesis.isw-bc.ca/

We have something like this for years in Poland and it works very well: http://usos.edu.pl/about-usos

There is where the federal government I think can help and step in to provide a hosted open source education system (wit a local appliance for caching and to survive outages). Not just for student registration and grades, but US-wide shared lesson plans, shared success stories, realtime metrics and for parents, etc.

it's open? where's the code?

As our funding is completely from the school district we currently have the code on our internal Git. The code itself is based on the Education Community Source license primarily so we can keep the development relevant to British Columbia. Once we have completed the product we intent to release the code to be forked by other jurisdictions to modify for their own purposes.

Please feel free to have a look at the openStudent website for documentation about the system and please ask any and all questions.

gregg

Reply on news.ycombinator.com