Dating is Complicated
by Larry Garfield
As part of our recent Balboa Park Online Collaborative (BPOC) project, we needed to push Drupal's date and time handling further than we'd ever seen it pushed. For those who haven't had to deal with it before, date handling is a nightmare. That's not a Drupal problem, but a calendaring problem. Date and time logic is incredibly complex once timezones, date-without-time events, and repeating events are added to the mix. What's more, there is always the ever-present feature request for "this event happens every Tuesday and Thursday this year at both 3 and 6 p.m., except for Thanksgiving and the 4th of July and this one other day where there's another event going on instead". Having implemented a number of calendaring systems of varying complexity, I've usually commented in the past, "Date handling sucks. It just sucks less in Drupal."
For BPOC, however, we had the opportunity to really do it right. BPOC really does have events like that, and we needed a clean, portable way to support them. Fortunately, there is an existing standard for such date logic that we were able to bring into Drupal: iCal.
iCal itself is a beast of a format. Not to be confused with the Apple application of the same name, it is a text-based format that was a compromise between a handful of existing, competing formats for defining complex event rules. As such, it offers just a bit too much flexibility to be called "easy to use", despite being the industry standard. We needed a good way to round-trip data from iCal strings to data structures in Drupal to all of the different instances an iCal recurrence rule would generate. No such code existed in one place, but we were able to assemble a solution.
There appears to be a severe shortage of good, complete, maintained iCal solutions for PHP. We found several partial attempts at libraries that stopped development when they got to the hard parts and a few Web calendar programs that used iCal internally, but didn't offer a separable library or API for our usage. Eventually we settled on iCalCreator, a partially-abandoned PHP 4-era library that handled more of the iCal specification than anything else we could find, but didn't hide iCal's warts very well. It actually did almost all of what we needed with one exception: It cannot process RRULEs, the iCal way of defining when and how an event will repeat, to generate each date instance we need. In fact, we found exactly one implementation of that logic anywhere: In Drupal's very own date_repeat module, part of the Date suite. How about that?
Enter the iCal module, which leverages the Libraries API and Autoload modules to stitch together iCalCreator and date_repeat and provide more solid Drupal integration. It also makes use of PHP 5's powerful iterators and DateTime object handling (which is, actually, really good) to provide a good, clean, predictable API, at least for the parts of the iCal spec that we needed.
Now that we could process iCal, what do we do with it? While the date_repeat module already exists, it was insufficient for our needs. It didn't allow each instance to have custom data and we expected that CCK wouldn't scale well in practice to having 500+ instances of a single field. In order to support both of those requirements we had only one viable alternative: Every instance of a repeating event is its own node, and refers back to an Event node that ties them all together and holds the original iCal information. That would mean thousands of nodes, but that's OK. Drupal can handle thousands of nodes.
That required creating three more modules: icalfield, icalreferencefield, and inheritedformatter.
- icalfield is a field that holds an iCal fragment, specifically a "VEVENT", or event record.
- icalreferencefield is, as the name implies, a nodereference-like field module that refers not to a node, but to a specific value within an iCal field on another node. That is, it refers to the recurrence rule that spawned it.
- inheritedformatter is an odd little formatter that lets an empty field pull in the value from the same field on a different node, "inheriting" its value, essentially. It works with both nodereference and icalreference fields and has hooks to add more.
Now, when an Event node, containing an icalfield, is saved, we use the iCal module to generate the appropriate Instance nodes along with it, each with a single-value date field representing that specific instance, all of them referring back to the Event node. Those we can then show in calendar views, edit individually, rate, sign up for, comment on, or anything else that you can do with nodes.
There's only one problem: Do we really want to generate the next 3 years worth of daily events all at once? Probably not, especially when iCal lets you specify events that repeat forever. Drupal can't handle Aleph-Omega nodes. Instead, we only generate out up to X instance nodes at a time, up to Y time in the future. (Both X and Y are configurable.) Then we keep a list of all "active" Event nodes and periodically toss them into a queue. That's right, the Drupal Queue module, which is a backport of Drupal 7's new queuing system. We then process each event again, generating any new instances. Over time, we will always have up to X instances Y time in the future generated. Because we're using the queuing system it should scale almost indefinitely for calendar-heavy sites.
And there you have it: a single place to edit arbitrarily complex repeating event rules, but a separate node for each instance with all the power that it brings, and it scales.
The next step, of course, is to clean up that iCal part. PHP needs a really solid, modern PHP 5-based iCal library, and we'd love to be able to move over to that rather than stitching together various different bits and pieces. If you're interested in working on or funding general PHP date handling work, let us know. We've been inside iCal and lived to tell about it.
Comments
iCalendar vs. iCal
You pointed to iCalendar specification but keep referring to it as iCal. They do not share a name.
iCalendar is the name of the specification, it says so right there on the RFC. iCal is a product from Apple. They have similar names, but not the “same name.†I've noticed people confusing this since about 2002.
This sound brilliant. Will
This sound brilliant. Will the other two modules also be released on drupal.org?
Btw, the link to inheritedformatter is incorrect.
They are
icalfield and icalreferencefield are both part of the ical module suite on Drupal.org, as are a couple of others.
And bah on the broken link. Fixed now. Thanks.
Impressive!
Not exactly an out-of-the-box solution yet, is it? In addition to icalfield and icalreferencefield, there's a fairly hefty glue module holding it all together and doing the queuing?
Also: Correct link for your http://drupal.org/project/inheritedformatter which is crazy cool.
Some assembly required
At this point yes, it is still "some assembly required". However, it's not a ton of assembly. For the client we wrapped it all up into a couple of feature packages along with other configuration that we'll cover in future posts. The glue module, icalinstance, isn't actually all that hefty. Not compared to the ical library itself. :-)
However, with the ical suite as released right now you need only create two node types, add two fields, and configure one rule. Done. I'm actually rather surprised how well it works. :-)
Yeah, sounds great!
Karen has done an amazing job with the Date & Calendar modules, but just as you said, all the last details get really hard.
Your approach above looks great.
Will all related modules be contributed? How does it integrate with Date & Calendar?
I am just starting learning how to develop in PHP & Drupal, but I have become very interested in helping with the Date & Calendar modules. Karen recently made me co-maintainer of Date & Calendar.
There are LOTS of issues & feature requests that it sounds like you have solved above, especially regarding repeating dates.
Works with Date and Calendar
The setup entails using the Date module and it will work with Calendar module.
Each instance of an event is a node and that node has a date field on it to specify the date/time of an event. This should work with calendar just like any other node with a date field on it as well as with the Apache Solr Date handling.
We went beyond just integrating with the module and are using some of the Date modules internal APIs rather than duplicating code. Larry already eluded to the use of the code that turns RRULEs into Date objects. We, also, utilized other functionality such as the time zone handling.
All the code is released
Yep, all of the code discussed above is part of the ical module suite on Drupal.org. (Except inheritedformatter, which is its own module, also released.)
And Matt, I eluded nothing. I did, however, allude to the fact that the only working RRULEs parser we found anywhere was in date_repeat. ;-)
We tried very hard to ensure that this was a "Drupal integrated" solution; as Matt notes (and I may not have made clear enough in the original post), the instance nodes have a normal single-value date field on them. Anything you can do with "nodes that have a date field on them" you can do with instance nodes: Put them in a Calendar, make a View, make a custom View, syndicate them (coming soon), use Deploy on them, use node_access rules on them... It's all there ready and waiting. That's why we built it the way we did.
Ah, Great!
I REALLY need to check this out and kick the tires... now to just make some time.
Could you provide a Feature for this so it can be better shared and distributed?
I am sure I will have some more questions and maybe some suggestions later.
Thanks!
Genius!
I struggled with this on my very first Drupal site. I ended up using the Date Repeat API and Date Repeat Node Generator.
Now to get this all wrapped up into a Feature!
Awesome work
Terrific work. In hindsight, how could it be done in any other way :). That’s how all brilliant advances seem.
It is stunning though how many dependencies are in play for repeating dates. The project page lists five and that omits the giant CCK/DateField+DateAPI projects. Justing pointing it out - nothing needs fixing here.
Hope someone can port this to 7. Don't elude D7CX.
Complex
Yeah, the dependency list is unfortunately nuts. I think I forgot a few, too. The alternative was duplicated effort, though, so we figured it was a good trade-off.
Due to the heavy use of Javascript and Ajax in the UI (the subject of the next post in this series, stay tuned), a D7 port is going to be a significant undertaking. We also have discussed moving from Nodes-with-Fields to making Event and Instance their own entity types. That would simplify a lot of code; or at least has the potential to do so. We're hoping to find a client who needs this functionality in Drupal 7 enough that they can fund a port and architectural improvements based on what Drupal 7 lets us do.
Anyone out there need a complex Drupal 7-based calendar? :-)
Dates sounds simple, but they are not
Great work and write up!
Dates are very complex, especially when you include all of the things you had to worry about. What makes matter worse is the clients expectation that dates are simple, both from a programming perspective and from an admin UI. I'm curious to know how all of these different elements are controlled/managed from the UI. The best interface i've seen (or maybe i've just gotten used to it) is on Google Calendar. Are you guys happy with the way your UI turned out?
I agree
Yes, Google calendar is definitely has the best online calendar ui I have seen. IMO, we would to well to learn from the google calendar ui.