An interview with Tom Dyson on Wagtail, the leading Django-based CMS used by tens of thousands of organizations including Google, NASA, and the British NHS.
Will Vincent 0:06
Hello and welcome to Django chat weekly podcasts on the Django web framework. This week we're joined by Tom Dyson to discuss wagtail, a leading content management system built upon Django. I'm joined as always by Carlton Gibson. Hi, Carlton. Hello, and Tom. Hi, everyone. So before we dive in, Tom, do you want to do a quick background on wagtail? And how you got into Django?
Tom Dyson 0:26
Yeah, sure. So I'm, I'm one of the two founders of torch books. We're a digital agency based in the UK 70 people now split across two cities, Oxford in Bristol, but we've been going for what long time we started in 2000. And we've been using Django since really the early days. So 2008 I think I made the first ever screencast for Django, which is still quite proud of it. You can look that up and
Will Vincent 0:52
what year what year was that?
Tom Dyson 0:54
30,000 5006 2007. Maybe I'll send you a link
Will Vincent 0:59
okay. Yeah, one
Tom Dyson 1:01
thing is that I picked the same bit of Django Reinhardt guitar for that screencast that you use for Django chat.
Will Vincent 1:08
So no one has mentioned it, because it's Yeah, it's sort of the Django Reinhardt. Jingle. And yeah, I always even. Yeah, I wonder if listeners picked up on that. But you did. So maybe you're the first,
Tom Dyson 1:20
I'm sure many others to do. So. So yeah, we've been we've been big fans of Django for a long time. In fact, Andrew Godwin, who I hired to serve as a fresh face university student in Oxford, where we're based, and he joined touch box. And it's actually for one of our client projects that he wrote South which, of course, became the migrations framework, which is now part of Django. And Simon Wilson, one of the two authors of Django also worked here for a while on a project called the carbon account kind of ambitious, but ultimately doomed project to help companies track their carbon footprints in a very precise way. So, so I feel like we've got a bit of Jango here. heritage that we're very proud of.
Will Vincent 2:02
Yeah, I'm impressed. I mean, I know Django was created in Kansas, but I mean, Simon, who's British was was there. I'm impressed by always the large English footprint. I guess as you know, as an American, I sort of assume the world revolves around the US. But there's a large number of very important Django, folks. I mean, both of you, Tom Christie, Andrew, Simon, you know, I wonder if there's something specific about England or that's just, you know, it's a global programming community.
Tom Dyson 2:32
You know, I don't go. I was struck by that as well, actually just in Copenhagen and Django con Europe where Colton and I worked last weekend. Together did seem to be kind of a high proportion of British people there. And I'm not sure the reason.
Will Vincent 2:48
Melton, do you have any Any guesses?
Carlton Gibson 2:51
Yeah, so Yeah, I do. I mean, like so I don't. A British people probably noticed British people that so there's a might be a slight bias in our observation. But in America, in Europe and Australia, we have been, I think, you know, if you look at the greatest hitters, there's it's not just Britain, it's Europe as well continental Europe, America, Australia is, you know, we're trying to make the that that contribution base more diverse, but historically that's been, that's been where it's at.
Will Vincent 3:20
And then Tom, I believe so wagtail, you, you were at the agency, you found a way to convince a client to help fund its initial generation, is that right? That's right. So we've been,
Tom Dyson 3:32
I guess what we're best known for in the last 15 years is building big public facing websites, content, manage sites, generally for for people making the world a better place. And we in 2007, we adopted Drupal in very well known PHP open source content management system that you know, has got a lot of momentum, particularly in that nonprofit space and fantastic community and very, very powerful piece of software. And we build a lot of sites on Drupal that will pretty proud of big organizations and, but we just increasingly found it a technology that was hard to love. And at the same time, we're building apps in Django, and just really, really enjoying the kind of the acceleration that we got from from using from using Django and Python. And, and I guess we didn't, we didn't set out to write another content management system, because it's, you know, it's, it's a kind of, it's something that you you hope somebody else will do for you. And, and there are lots of great tools out there. But we were commissioned by the world College of Art, very prestigious arts university in London, who themselves had done a big valuation of content management systems and hadn't found quite what they wanted. And so it was an opportunity for us to, to build something that we that we felt was going to learn from the lessons of using other systems on our favorite framework on Django, and that was in late late 2013 and Soon afterwards, we we open sourced in February 2014.
Carlton Gibson 5:04
And was that always the plan to open source? It was that like part of the agreement, part of the contract to build that?
Tom Dyson 5:10
It was part of the contract. And we were very happy that that that was, you know, a requirement. And we've been big supporters and users of open source over the years, but I guess we hadn't anticipated. You know, because there's, there's different ways of open sourcing things, you, you can just build your project for your client, and then take out the secrets and put it on GitHub. But but we wanted to do a bit more than that. But I guess we were slightly taken aback by the response that that went ahead. And we hadn't we hadn't anticipated that it would quickly become a popular tool. And as of last week, I think it hit 7000 stars on GitHub, which I know is, you know, it's just one metric among many, and it's, you know, a bit of a kind of a vanity metric, but it is, it's I think that's I think that means that after Django rest framework, it's The biggest Django project
on GitHub.
Carlton Gibson 6:04
Yeah, I think that's possibly probably true.
Will Vincent 6:06
Well, I think I saw that. I mean, in terms of people using it, I mean, there's real college, College of Art, I think Oxford, the British NHS, NASA. I mean, it's, I was I was impressed by how widespread its usage was.
Tom Dyson 6:20
I think I think people always, you know, we we put those big names up at the front because they give reassurance and that was protect particularly important in the early days of work to win for our clients, some of whom were quite risk averse, then they could they could see the benefits from the UI point of view. And they they believed us that it was going to be quicker to build their sights, but they were nervous about something that they hadn't heard of before. And but now being able to reference Google and NASA and the NHS makes a big difference in particularly in the UK, the NHS has been, you know, that's a really big deal because the National Health Services I think, still the world's fifth largest organization, and this is not just a sub site. This is the main NHS site is nhs.uk and They did the migration from massive Microsoft infrastructure in a very public way. So they, you know, really happy to speak about it. And it's um, you know, this this this is a site that's very well known to anyone in the UK and it's like the first place you go to if you're worried about the curse strange and you marks on your arm or something. And, you know, and, you know, it's an important public sector site. And I feel like this is it's kind of a it's a big story, not just for whitetail, but also for for Django and bison.
Carlton Gibson 7:33
Yeah, totally, totally. Early sites, because people always say still, Django doesn't scale that you can't build it. You can like these are these are some of the world's biggest sites built on Django. And yeah,
Will Vincent 7:43
I mean, NHS Instagram, it's pretty much as big as it gets. So I so Carlton, because you're, in addition to being a Django fellow, so involved with Django rest framework, I hope we can talk a little bit about what it's like to run a hugely popular, open source project that relies on Git Because you have these dependencies, and I imagine there's a number of similarities in terms of keeping your own projects up to date, and then linking it up with Django, which has this aggressive reliefs, release cycle in part. Thanks to your work, Carlton?
Carlton Gibson 8:13
Well, many teams work as we discussed in the other episode, but
Will Vincent 8:15
yeah, be me because it all starts with Django, right? I mean, and so with wagtail I mean, I think that's maybe something that was confusing for people who haven't used it is you can start a new project with wagtail. Or you can also just like Django rest framework, take an existing Django site, right and, and add wagtail on top of it. Do you have a sense, Tom, of which of those approaches is more common is that starting out initially with Mike Taylor, adding it onto an existing Django project?
Tom Dyson 8:42
I think the former is more common. So starting up won't help but we're really keen to stress that wagtail is just a Django app. And, and it sits alongside all your other apps. And it's it should be straightforward to integrate. And, you know, a lot of the time the confusion In in whitetail that we feel we can't really honestly take credit for because we're just standing on the shoulders of the things that you know that Django has already provided. And, and also, you know, the fantastic ecosystem that you get with with Django and Python means that if there are things that our clients want to do that that aren't available out of the box in whitetail, it's always, you know, a pinch all the way. Right.
Will Vincent 9:21
Yeah, I think I think you the release cycle is even more aggressive than Django. Right. I mean, I think I saw somewhere you would said every two months, but I think the latest was in December of 2018. For 2.4. Is that correct?
Tom Dyson 9:32
That's right. So we were more like three months at the moment, the target was two months. And I think I think we might revise that because two months maybe feels a bit a bit too tight. But this again, was in this is a guess in response to what some of the lessons we've learned with Drupal, and there was a long period between Drupal seven and Drupal eight, about three years and it was during that period that that upgrade cycle, it wasn't really clear how long it There was going to be, there were features that they wanted to build into Drupal eight, and you know, they weren't going to launch it until they were happy with it, which is, you know, that's fine. That's an understandable approach. But it means that if you're someone who's about to invest in Drupal or has an existing site, then you're left in a bit of limbo waiting to know, you know, should I wait? Or should I build on, you know, two year old technology. And then when it comes to to the upgrade, there's loads of improvements in Drupal eight, but the difference between seven and eight was so big that for many of our clients, the job of upgrading is comparable to starting again. And we really don't want to be in that situation. So we have we've strived to have a predictable frequent upgrade cycle. And, you know, we aim for two months and we've achieved that sometimes, but I think I think it's gonna be more like three months. But the idea is that when you do the upgrade, it shouldn't be more than 20 or 30 minute piece of work. And you're although the, you know, you're not getting a huge list of amazing new features every time because it's the cycle is shorter, you feel like you're just taking advantage in a more immediate way of all the work that the community is doing to make the software better.
Carlton Gibson 11:08
Because unship software is like inventory, right? It's like stuff you've got in the warehouse that you're not, it's not on the site. Now get it out, ship it, like,
just didn't like delivery. I like that.
Tom Dyson 11:18
As this this good, I'm going to use that metaphor.
Carlton Gibson 11:20
It's called spolsky. I wrote it. But the I think for Django that works exactly the same way like we've got the night is nine months more likely for Django, and but it's reliable. And it ties not just into the upgrade process that you've talked about making upgrades easy, but it's also about confidence in the supported versions policy. So they know you know, users know that their their version will be supported for that 18 months, and then they've got that time to upgrade a ton. You know, just everything this the whole ecosystem is more healthy. If there's a regular cycle that's, you know, relatively frequent.
Will Vincent 11:52
And is there still you do LTS is for wagtail? Is that still the case?
Tom Dyson 11:57
Yeah, we do. Yeah. And again, as the duration of those LTS is is kind of under discussion. And then we've had some funding from one big but currently secret sauce who has extended the duration of the LTS for their own benefits. So that's that's a nice a nice thing to do for the community. I think because you know, it having a longer LTS is an easy thing to ask for is easy thing to ask for. But it come comes at a cost for the maintainers who need to make sure that it's not just the current release that they need to handle, are there any security updates, but released from 619 months ago? And it's not, you know, isn't a huge consideration, but it's a there are it's one of the kind of small ongoing costs, that open source maintainer, so they have to be aware of
Carlton Gibson 12:43
also affects the users who are on the LTS thing. You know, there's a kind of LTS mindset that you you end up getting closer and closer to the end of life date and you still haven't updated whereas if you if you don't rely on the LTS If you can not rely on you, obviously it's not always possible if you can be in that Then you've kind of got a different attitude to updating, which I think is much more healthy for your product, not just right. Yeah, the dependencies.
Tom Dyson 13:08
But I think I've something I've learned in the last couple of years is that while it's, you know, we we can be idealistic about and, and make it as easy as possible for people to, to be part of a rapid upgrade cycle. It's just not always possible for the ways that some especially enterprise II organizations operate, and for them, it's, you know, the, the idea of, of having to assign even small amount of budget to an upgrade every two or three months feels a bit scary and they want to know that there's a there's a longer Lt. So I think that's something that we have to be conscious of.
Carlton Gibson 13:42
Yeah, I know. I do accept that. I just I kind of think that the same very same companies buy vans and cars and other motor vehicles and they don't object to allocating budget to servicing them.
Will Vincent 13:51
Yeah, Carlton, I try to be aspirational and push companies to be more active. This mind is still updating But I mean even I mean, Karl Meier has a Django under the hood talk. He's Django core, and then at Instagram and mentions even Instagram, which is maybe the Django site just went right from one three to one eight, and just saw what broke so, right? Yeah, it happens. Well, so I'm curious with wagtail. Because Carlton, I have talked about how at the end of the day, how small the number of people who really make Django happen is what's the sort of core size of contributors like how does that compare to Django.
Tom Dyson 14:30
So we have a, we have what we call the core team. And that's currently 19 people, five of whom are here, here at torch books. But, you know, I'm happy that five is no longer the majority of the core team. That was kind of a goal of mine early on in the project. I think in terms of the I don't have metrics, I guess I could look at lines of code in GitHub, but in particular, Matt Westcott here at torch box, who's the lead developer on whitetail is pretty much full time on on the open source project. So he's full time. And so I, I think in terms of lines of code, we're probably still in the majority. But I'm really pleased at the core team is growing. And the core team, importantly isn't just a back end serious back end developers, it's people now with really great UI expertise and people who can write documentation and people who are good at growing communities.
Will Vincent 15:27
Yeah, well, that's I mean, that's one of the things about wagtail is just the first look, the UI is is beautiful. And the documentation now fantastic. I mean, I know you've mentioned the past the documentation was something you've spent a lot of time focusing, I guess, on that, that first user or that first use perspective, right, rather than sort of the deep docs, that's more relevant to people who already know it.
Tom Dyson 15:47
Right. And I think there's still still much more we can do that. I'm really interested in that first, that first 30 minutes experience because I know that's the way that I I worked at other I explore new tech Apologies it's like in a lunch break, and maybe have half an hour and I want to be able to pip install or you know, get it running on my laptop and see something working. And then if that experience goes smoothly and feels good, then I'm excited to, you know, recommend it to one of my colleagues or look into it more seriously. And, you know, there probably is, maybe I'm too impatient, but i think i think i think that's not uncommon. And I think now I think that's
Will Vincent 16:27
standard.
Tom Dyson 16:28
Yeah. So I just think providing doing the best job you can in that in that first 30 minutes is really important. And that's just, it's not, you know, even in Python, it's not as simple as it could be, because we had a good comment recently, because we have this get started in seven lines, and why tonin starts pip install lacto and then someone raised a GitHub issue saying I, I couldn't begin to tell because he said I didn't have permissions and because we assume that you can do it in a virtual environment. You know, for someone someday might be someone who's interested in who's heard about Waltons interest in it and hasn't got as far in their Python development is knowing about that environment. So, so then we have elements of how do we do we need to explain environments? Or can we link somewhere? and and you know, actually there, there isn't even a kind of completely standard way of running a virtual environment now, source across operating systems. So that bits a bit tricky now, it'd be really good, I think in in Python generally to do better at that to support those those new 30 minute users.
Will Vincent 17:35
I think there's a there's a pep two, I think there's a new Pep to sort of remove, get rid of virtual environments in Python. Right. That's the idea. I think there's some work being being done on that. But yeah, I mean, I think I heard as well that when you first launched the project, your team would had been was using vagrant internally, right, and also made the assumption that everyone, I guess, that today that would be sort of Docker, right? Would Be the vagrant equivalent.
Tom Dyson 18:01
Right, right. Sure. And yeah, this is another thing that that we learned is that we, you know, we made, I think too many assumptions about the way that other people run, run Django projects based on our own experience. So I think we, we assumed PostgreSQL and a few other things in a way of running virtue environments, probably and, and so we quite quickly in the first six months started pulling those out, as we realized that that just wasn't a match for the way that other people were working.
Carlton Gibson 18:28
But it's probably not something you can predict April, right, you can try and explain it as simply as you can. And then you realize I'll actually there's hidden assumptions here, and it's hidden assumptions there, but you're not. The whole point is the hidden assumptions. You're not gonna be able to spot them in advance.
Will Vincent 18:40
So it's kind of okay, well, as long as you improve on it, Carlton? No, but like, Yeah, I agree. I'm agreeing and adding it's, it's, it's an
Carlton Gibson 18:49
ideal.
Will Vincent 18:50
Yeah, well, so what's the What do you think is the profile of a wagtail user because I could see someone who doesn't maybe know Django saying, Oh, well, this is an easier way to get into Django. Is that the case? Or is it more an existing Django dev who wants to, you know, have a CMS for clients? What would you say? Is the background? I think that's shifting.
Tom Dyson 19:09
And to start off with it was definitely this that that persona of someone who's a Django developer who wants content management for their for their app for their client. Yeah. And, you know, we want to be a good choice for that person. And we want to explain the differences between us and some of the alternatives. But I think as, as whitetails, reputation grows a bit, and then I hope that quota will start being an interesting CMS for people who aren't Django developers, in the same way that a lot of people who use WordPress, you know, I'm sure don't know, PHP or might not even know what PHP is, but we are still interested in using WordPress.
Carlton Gibson 19:46
Yeah. So the interesting question, then is the hosted solution because the vast majority of WordPress users they want to host like WordPress, not
Tom Dyson 19:54
on your own and also they want themes and that's, that's a really interesting question for us. One of our principles with Work tells designers that we should be completely opinionated about the kind of site that you want to build about the we don't we definitely don't want to inject any markup, for example, or make it make any assumptions about how you how you want to define your models. But the flip side of that is that when you run whitetail start, you know, you do the seven vines, and you kind of did Are you are you left with a, like a single homepage with no styling? And it's a it's an underwhelming experience for the first time you said, so. So clients finding a way to kind of bridge that gap between being unopinionated but also, you know, helping people start with something where they can understand what they could get is a challenge for us.
Carlton Gibson 20:44
Yeah. And still, you still have to kind of define your page models and, you know, create your own templates. And, you know, there's a lot of work to go from the, I've got it started and I've got the nice admin UI and all of that stuff to a fully featured site.
Tom Dyson 20:59
So So it's hard to imagine how what steps would be to from going from from that model to a hosted model where where wagtail becomes, you know, something that's like a better option for someone who, who doesn't yet know about creating models. And one, one approach I've been thinking of is something like, you know, so if you're building a react app, now, you probably use this create react app, create a react app, command, and that that, you know, will start out an app for you and you can continue using it and then at some point, if you want, you can eject from that from that model. So maybe we could have something similar where you, you can you can have a kind of quick start, I tell that that has a blog and, you know, custom user model or something or some some of the basic things that people want to do. And then you can object out of it and and write write your own models and migrations.
Carlton Gibson 21:52
Yeah, I saw one extension to wait Oh, it's I think it's called Code Red CMS, where it's very much designed for creating landing pages and marketing sites. Real quickly, and it uses bootstrap for and it's got some predefined models. And yeah, it's kind of good if you want to build that site. It's that kind of site. It's it's very quick and good, but it's built on top of wagtail. It sort of made some of those decisions. But underneath underneath the hood, it's, you know, you've still got
Tom Dyson 22:15
a wagtail process, right? Yeah, it's a really interesting project code, right.
Slightly unnerved when I first saw it, because it seemed to be, you know, so new CMS using my toe, but actually, they're quite clear in their intentions. And I think they, they expressed the differences in a nice clear way in the, in the readme for the GitHub. And it's just like you say that you can, you can build a simple marketing site using Code Red really quickly. And by you can theme some bootstrap templates and, and then if you feel like it's, you need to customize it or extend it more. You can kind of eject out of that and just treat it as a normal web site.
Will Vincent 22:50
Yeah, maybe let's let's talk about some of the features in wagtail because I think you alluded to it. It has a lot in it and when you just fire it up for the first time and see the the simple home Page, it's not really clear why you would use it. But there's I mean, we can go down the list, right? I mean, stream fields, search images, I mean images in particular, right? Because that came out of the first client, the idea that you can have focal points and automatic thumbnails. I mean, these are really amazing features. I kind of wonder what the process is where someone finds out about them. Because I mean, the documentation is great. But there's someone I wonder if there's like a killer feature, in your opinion that brings people over versus maybe once they know wagtail, they say, Oh, this is kind of keeps me here. I didn't I
Tom Dyson 23:32
didn't know this. I think maybe the killer feature in the early days was that the UI was was kind of looked good and was thoughtful and was really focused around the native editors. And, and I think you know, that there's lots of wonderful open source software out there, but we we had the benefit when we were building wagtail have some some fantastic Python developers, but also a UX team and designers who and when we took the time to to think we About the the editor experience and making it kind of responsive and seamless and thinking about editorial workflow and making it look good. And so I think that was probably the sort of the killer feature to start off with. But the reason I think that once we once you once you started using wagtail, I think this stringfield feature is probably what, what what stands out and stringfield is, so if I was going to describe it in terms of other content management systems, you have, you have two kind of two possible models, probably very simplistic way of describing it, you have the model, which probably more natural to people to Django developers, where you define all the fields that you think your your, you're going to need to, to hold your data. And so you have this really nice structured data and it means that you can filter it and you can look at your keeping a really nice clear distinction between presentation and, and and data. On the other hand, you have tools like WordPress Whichever really popular because they're easy to use, and you have one big field and you can you type the body, and then you can use kind of wizard type editor and copy and paste word stuff into it and have tables and inline images and so on. And it's easy to use, but you're you're kind of munging up the presentation and the and the data. And that means it's difficult to export that in an API, for example, to a native mobile app. And, and so there are pros and cons to both approaches. But an stringfield was our,
our,
our design for stream field was to, to try to bridge this gap to, to maintain structured data, but to give editors flexibility within pages. So you can have blocks that repeats and are reordered, and they're optional. And so you can create, like, more interesting narrative type long form content. And, but but it's still stored in a structured way using JSON in the back end.
Will Vincent 25:58
Yeah. So you Have a header
Carlton Gibson 26:00
block and a text block and an image field block and, you know, build your page up from these.
Tom Dyson 26:06
Yeah, yeah, exactly. And and I should say this is this. I think this was a pretty fresh feature when we launched it. But it's not it's not unique anymore. No, there are there are implementations of this and other systems. In fact, I think the new WordPress has a has a quite a good implementation of this, but I think it's a it was really, it was well designed. And I think my colleague, Matt Wescott has to take a lot of credit for that.
Will Vincent 26:29
Yeah, I think that makes a lot of sense if you've had experience and frustrations of the other two approaches, whereas it may not seem quite as impressive for someone new to Django, but certainly for me, that makes all the sense in the world not to get locked into that. Yeah, one big block pattern. Yeah, that causes frustration down the line,
Tom Dyson 26:48
but it's not exactly it's not a kind of killer sales feature because it's, you know, took me five minutes to explain that to you. And it's a it's it's feels like what is it a subtle point, that ends up being important, I think, but
It's not you know, it's not the kind of thing you put at the top of the list of your, your core features. Perhaps you just need to work on your pitch there. I think I don't because it's quite cool. No, I think
Will Vincent 27:10
it is to just say, here are the two dominant approaches, and they both have drawbacks. But it Yeah, it is something you need to kind of feel the pain first, you can't just trust the pain. So well. So images to I was really impressed by all the, you know, images, right? This is are made you want to give that pitch on all the built in image features.
Carlton Gibson 27:29
Well, just before you do that, for me, it was amazing. Because in Django land when wagtail came along, there wasn't you know, there was some third party apps out there that handled image uploading, but they none of them were particularly nice. And then wagtail did it really well. You know, you could upload images and as you say, slick focal points and all these kind of things. That's actually not very glamorous, but it's quite hard to implement properly. And wagtail kind of did and it was like, yeah, this is really nice. That's nice to hear. But
Tom Dyson 27:55
I think I guess it was partly driven because our first client will You've heard as you imagine, you know, the visuals are really important to them. And, and so we needed to, we would make sure that they were displaying the image content in a really clear way. And also something that I've just noticed a lot in some pretty big, high profile news sites, you know, you often get this situation where you have on the on the details page, so like a news item, you have a nice big hero image, and maybe the photographer is, you know, like the rule of thirds, and the, the, the portrait is, you know, over on the, on the left or the right hand side, but then on the Index page showing the latest items, they'll do like a square crop. And, and sometimes the crops are, you know, cutting off off the face of the person to subjects about and so that was in Waikato, you specify the focal point, the focal point. So you draw an area around the thing, which you want to crop in. And I think that's more flexible than allowing editors to do actual crops because you don't know quite how that focal crop Use difficult one might be used in in future designs. So it means that then, you know, when you're resizing the image for different devices, for example, it's just going to make sure that it's always maintained around that around that point. And I think it feels like a small detail, but it's, it's, it can be the difference between a site that looks uncared for, and one that looks like it's really making an effort for its audience.
Will Vincent 29:21
Yeah, I think that's really important, especially since Django is a predominantly back end framework that that love and care for the front end. Yeah, matters and matters a lot, even to the most, you know, design challenge back end engineer, they can tell when it's thought out
Carlton Gibson 29:34
forgotten, like the whole point of a CMS is that you take the the sort of the hand cropping element of the work out of it for people, right that you don't want them hand coding HTML, you don't want them hand cropping images. So to have it done automatically and nicely. Is is the raison d'etre.
Tom Dyson 29:52
Yeah, I agree. And actually, there's been some really interesting work in in the kind of third party ecosystem around innovation. handling. And one of my favorites is by this, someone from a Swedish agency, Martin Sandstrom, who he built a plugin called wagtail fortify. And this uses these hosted machine learning tool. So computer vision services, and the one that was best actually the Microsoft one. And as you upload your images, it will send those images off to the to the image recognition service. And within, you know, a few hundred milliseconds, you get a suggested title and tags for each image as it's uploaded. And this was in your Django con Europe talk. That's right. Yeah, my talk was about was not about well, it was about how Django developers can can take advantage of these cheap, amazing Cloud Machine Learning Services and sort of inject magic into their apps. And this is this is one really, really nice example of that, I think, and it's not you know, that You always get a laugh when you show this because some of the some of the descriptions are obviously inaccurate, but actually, you know, they get better and better. And I think the expectation is not that you just allow the machine to tell you the answer, but you use it as a as an eight. So it's, it's something that should augment the editor experience for particularly for large, really big, busy new sites, I think. I think it's the kind of thing that can really chip away at the sort of the, the, like, the basic legwork that editors have to do.
Will Vincent 31:32
So what's the process by which something like that makes its way into wagtail? I mean, like, South is a great example of, you know, migrations coming into Django. Is there an example of that? Or do you just really focus on wagtail itself and just let the third party be third party since that's a lot to manage?
Tom Dyson 31:48
Yeah, so that's an example of something that isn't in wagtail itself, but but because, you know, it's, it's really cool, but we don't imagine that everyone wants to use that. But we we highlight it, so there's a there's an Awesome whitetail, you know, the, you know, these awesome x repositories that
Will Vincent 32:03
have the awesome Django one?
Tom Dyson 32:05
Oh, wow. Well, congratulations.
Will Vincent 32:07
So there was an older one, but it stopped being maintained. So I have the new one uses more awesome. And
Tom Dyson 32:14
someone made a, an awesome octo and we keep that up to date. And you know that that's, that's currently where we we highlight the kind of community plugins, but actually there's been some discussion about whether we should do a better job of that. And you know, there are a few apps that have their own kind of nicely presented marketplaces. Not that I'm not I'm not really imagining a kind of commercial marketplace. But but something that makes it easier to surface these kind of tools.
Carlton Gibson 32:39
Yeah, because ecosystems now got the size to the size where it's discoverability. Difficult, right? You can't just follow one account on Twitter and
have it all appear. Yeah, no, I think that's true.
Will Vincent 32:49
Yeah, why not? That's a challenge for Django itself is, you know, Django itself doesn't have an awesome Django repo because then they're sort of being opinionated and, you know, frankly, as a book author, I wish they had Books page. But they just listed all the Django books because there are very few. But you know, when someone Google's best Django books, they find my blog post instead. So I understand that tension
Tom Dyson 33:11
isn't isn't one of the rules. I don't know if it's an unwritten rule, one of the rules about the awesome x repositories is that they can't be started by the creators of x.
Will Vincent 33:19
Oh, I'm not sure well, so to be frankly, honest, so I started the awesome Jango repo again with the idea of, Oh, this one's out of date. And then I started down the process with the awesome maintainer of going through all the steps needed to do and then I just got swamped with stuff. So it's technically not in full compliance. But I you know, my way of maintaining it is I just say this is basically wills repo and, yeah, you're welcome.
Tom Dyson 33:48
Do you accept submissions?
Will Vincent 33:52
I should people have done them on occasion. I'll accept them, but I don't feel bad about just making it pretty arbitrary because the problem is they just bloats over time, as you just add more and more you want to take away and I mean, I know a lot of the people too, so I don't really want to be. So it's a tough thing for any awesome or any repo is being you know that referee position, but Eurasia is nice. I mean, yeah, for example, like there's third party packages, there's the Django packages site, which lists all of them and you can filter a bit. Awesome. Django has, I don't know. 20 it's, you know, it's not like strictly by stars. It's basically the ones that I think are cool and I'm constantly adding new ones but you know, I was getting kind of burned out on awesome Django thing. And I was just like, you know, I'm just gonna do it for myself. But yeah, that's there's no perfect way to do it. So no,
Carlton Gibson 34:46
but it's really difficult to feel the requests like so you asked earlier on about maintaining a large Oh,
Will Vincent 34:52
yes. Tell us Carlton's exactly
Carlton Gibson 34:54
that no, but it's exactly that problem is there's ever coming in streams. Can you add this Can you add this? Can you add this and you have to choose otherwise, it becomes unmanageable. I think the the issue, the Django Doc's has a policy look, we just don't link to third party things, because they disappear. And we have to keep them updated. And they're not reliable. So we need sites like an awesome Django or awesome wagtail to curate the list, but it has to be curated. Otherwise, it's just noise again.
Will Vincent 35:22
Yeah. So Tom, if you could snap your fingers, what would you like to change? Or what features Would you like to see? You know, what perfect developer drop down and implement in wagtail?
Tom Dyson 35:31
I think it's a I don't think so. A kind of a brand new feature. But I think the really important direction process supporting the headless model better. And I think this is a clear direction in content management generally. And we're seeing more and more of our own clients wanting it and talking about it. And, and and, you know, just more generally in the industry. There's some really interesting products and services, things like contentful and propriety hosted headless CMS. And
Carlton Gibson 36:02
so this is where the CMS acts just just exposes an API, which then clients used to build sites or
Tom Dyson 36:08
whatever. I should have explained that so with
Will Vincent 36:10
like react or view or Elm, Carlton, right,
Carlton Gibson 36:14
or even a even a publishing pipeline might consume content from the CMS, if you're creating physical newsletters or something, but the idea being that it's the CMS kind of stops at the API level.
Tom Dyson 36:25
Yeah, it's not, it's not responsible for the presentation part.
I was gonna say though, whitetail actually supports this pretty well. And really, really early use. In fact, probably the first user vote on this way was the Hillary Clinton campaign, who, who kind of secretly commissioned the first API for wagtail in order that they could use it in a headless way. And, you know, the people who were behind the Barack Obama campaign, kind of a lot of them made their careers out of the success of so we were waiting excitedly for the day that we're going to say that.
Will Vincent 37:02
Yeah, that a colleague who worked on the Hillary campaign and all right, I remember him mentioning Django, and I was like, What? Now? He was sort of hush hush about it. And all that ties that circle together. Yeah.
Tom Dyson 37:15
Anyway, that story didn't didn't quite play out. But yeah. But
Carlton Gibson 37:20
there's always 2020.
Tom Dyson 37:22
So they commissioned the first first API and so that that was the head of the site and, and then actually, Tom Christie, the, you know, the author of Django rest framework, rewrote the whitetail API to use DRF, which is, you know, that was, that was a good day that that PR landed without warning, notice just popped up in GitHub. That was pretty exciting. And so, so worked out works well in that way. But we don't do a great job of documenting it. And I think we should have some good demos and blog posts and there are some features that we can add. So in particular, previews is a bit of a problem with With headless CMS is generally because, you know, in standard workflow, when you hit preview, we're able to take the data that you've just been writing and push it into the into the template, and you immediately see how it's going to look. If you're,
Will Vincent 38:14
if you're an editor mode, so not on a live site. Yeah,
Tom Dyson 38:18
exactly. You're in editing mode you're seeing and it's not even draft, it's, it's just the changes you've just made even before, you may have saved it as draft. And you really want that cycle to be to be quick. And you know, to first read as opposed to kind of play with their content and immediately get feedback on how it's going to work. And but if wagtail isn't responsible for the front end, it's harder for it to represent that. So we we want to find a good generic way of being able to preview from from different front ends. And that requires making some changes to the API that mean that we can store that. That kind of ephemeral preview data in the API and whether it's accessible on the API which has its own challenges. Because then the front end may have it needs to authenticate in a different way.
Will Vincent 39:04
Yeah, authentication is the one that jumps out. Well, when you look at changes in Django, I mean, specifically, I guess, async. Are there any things down the line on the roadmap that will cause will force major changes within wagtail? itself? You know, 3.0 and all the rest?
Tom Dyson 39:19
I think, I think async is the only one and async as we've just I mean, it's really exciting for us and for all Django products, I guess, because because of the potential for rapidly improved performance.
Will Vincent 39:36
Are you a big believer in that? Because I know, yeah, there's, you know, so it definitely can work in some cases. And at massive scale there, there is a question of whether it's, you know, more than you need for a moderate site, I guess. Right. I'm curious what your personal
Tom Dyson 39:49
I'm a believer in the potential performance gains. I'm I guess I'm a bit skeptical about what it's going to take for the Django and the Django community to adopt it in a way that realizes those gains.
Carlton Gibson 40:05
There is there is secret news in that Andrew Godwin opened Yes, pull requests on Django master last week at the sprints, right hunger con Europe Sprint's introducing like that that first step into an async handler at the very HTTP layer. And that's not the middleware. And that doesn't let you yet write async views. But that's that's definitely manageable, not maybe exactly is but with a review, that's definitely manageable. And then that's we're on the road. And it's coming, I think, where the the async really could help a project with like wagtail is where you've got to do multiple fetches. And a REST API will pick one model and then another model in another model, it would be very easy if we had async views to write a kind of proxy view that did a couple of fetches and combine them without necessarily having to jump all the way into say graph qL or something, you know, which is a totally different technology. Exactly. Yeah. So I'm example I'm
Tom Dyson 41:00
excited about I think we got another another relevant use case in wagtail is cache invalidation. You know, one of the one of the hard things we, we worked through plays nicely with a lot of the caching proxies like CloudFront, and CloudFlare, and varnish, and so on. But those those invalidation calls can be quite slow. And we, you know, you want to check what point invalidation has successfully returned. So the simple way is to run celery and have a worker and keep checking. But if we can just do that in an async way, and that just simplifies a lot of code.
Will Vincent 41:34
Yeah. And yeah, so another feature that with wagtail, I guess it's an experimental is the A B testing framework. Can you briefly talk about that? Because that looks really interesting.
Tom Dyson 41:44
Yeah, this is, it's, again, it's not part of itself. It's, although it's under the Watchtower organization and GitHub. It's, um, it's a tool. It was commissioned by a bank in in Norway. And, and that's relevant because there are so many great products and services for doing a B testing and one that a lot of people know about is Google Optimize. And, and that works by, it's in a JavaScript that you embed. And then you have this UI that's off site that, that lets you define the different variants. So the different options that your users might see. But it means that when you load the page, the JavaScript is having to rewrite bits of the DOM. And, and as for, for banks, and this is definitely nowhere and probably I imagined in lots of lots of territories, it's not allowed to have third party systems rewrite the content of the
Will Vincent 42:36
of how interesting yeah, it's probably, yeah, the US very draconian with how they
Tom Dyson 42:41
Yeah, right. So and rightly so. So that that rules out, you know, the use of these these kind of sophisticated front end tools, and so they commissioned us to build something that was work natively in Whitehall. And, and it's, you know, it's definitely simpler than those other tools which you know, you know, have gotten to Are people working on them full time, but it allows you to create different variants of your of your content, usually as different pages, and then to start running experiments, and then it uses middleware to swap between them and typically setting a cookie to make sure that you that users are directed to the relevant one each time. And then as some simple reporting, and and it's not, as I said, it's not it's not the most feature for a B testing but I think the important thing about AV testing is just to start doing it you know, everyone knows that it's it's best practice that you know, especially if you're getting reasonable amount of traffic, you should just test which option results in more donations for your for your charity, or signups your letter or whatever it is, or sales for your product. And, and then the tools are there for you to run those experiments and pick the best one and then move on to the to the next so the our focus has been on having something that works simply And is easily integrated.
Carlton Gibson 44:02
So from a web web performance point of view, like having JavaScript rerender the DOM that's going to be quite slow, right? noticeably slow on mobile devices. And that's right. And whereas having it rendered services is going to be, you know, the same speed essentially as not doing it. Right. So from a performance right,
Tom Dyson 44:18
and, and there is still you know, we kind of old timers remember the the so called flash of unstyled content that we used to get, and, as I said before, the CSS kick kicked in, and, and you still get the flash on an AB tested content with these tools, which you can minimize in some ways, but it's it's a bit tricky to manage. There are still performance implications, though, because if you're doing a B testing server side, that makes it harder to do straightforward. caching using CloudFlare or CloudFront. On one of those two. Okay. Yeah, fair enough.
Will Vincent 44:49
Well, you know, as we sort of wrap this up, Tom, one question I had for you is, since so much of your work is with nonprofits, and you seem very mission focused. You know, I saw in a talk you mentioned the moral dilemma of, you know, open source software where wagtails use by lots and lots of people doing lots and lots of things. Yeah, I wonder if you could just speak to that. Exactly. You know, Django is used by lots of lots of sites, porn sites, all the rest, what your thoughts are on wagtail doing that? Because most licenses you can't, there isn't really an open source license where you can restrict the usage, I don't believe
Tom Dyson 45:21
no, I don't know what it is an interesting dilemma. I don't know if it's a dynamic because there's not much we can do about it. But it's, there are definitely people using wagtail who who we wouldn't work for. Yeah. And and also the question comes up about you know, and and some of those May, some of those people may be asking questions on the on the whitetail slack or, you know, Stack Overflow and so on, and we're committed to helping everybody in this community so so yeah, you know, that we definitely will be by one way or another supporting people whose whose goals, you know, we may not agree with, but it's pretty hard to think to try to you know, forensically. Yeah. On the who's on the right side of the line. And I guess we generally feel about open source as a, as a force as positive force in the world and, and people using open source. So, you know, in in a small way, helping with that. And so, you know, there's there's a kind of justification for a guess for supporting open tools generally.
Will Vincent 46:23
Yeah, no, I just I, what was interesting to me is just that you asked the question, because it's not something often asked. And certainly, it should be in technology. So I just thought, and I guess that comes from many of your clients and running an agency and choosing who you work with. That makes a lot of sense, right?
Tom Dyson 46:38
Yeah. I'm sorry, I don't have a more coherent answer.
Will Vincent 46:41
So was there anything Tom or Carlton we wanted to include as we as we wrap up? Well, what's the what's the future of wagtail? Right, like you sort of alluded to it was good. You know, if you look ahead, you mentioned headless. Potentially themes and deployments are there where Do you like to see it in a couple years if it all plays out?
Tom Dyson 47:03
I think the two main directions are the ones I've mentioned. So one is making wagtail feel like a attractive option for for for anyone, not just a Django developer. And I think a lot of that is about, about improving that, that experience of the first 30 minutes, which could be having a hosted service or just really, really working on that on those, those beginner steps. And the second is the headless thing, I think, you know, kind of got quite clear goal for us could be to be the top choice for open source, headless CMS.
Will Vincent 47:35
So yeah,
Tom Dyson 47:36
that would be whatever, whatever the technology if you if you if you if you want to go ahead, listen, you know, you've got great commercial tools like like contentful, but if you want to go open source, then I'd tell us your top choice that that would be good. Something for us to strive for, I think.
Will Vincent 47:51
Yeah, I know Carlton, I have talked to a bunch about deployment in Django because that's, you know, as the as the author of books, I sort of I get some Any questions about the deployment sections, which I use Heroku, which is, you know, as easy as it gets, but it's still quite a leap. And so I often also think about how do I you know, can you just push a button and solve this for people? Because it is it's really tricky even with platforms the service
Tom Dyson 48:16
Yeah, yeah.
Carlton Gibson 48:17
It would be I mean even with companies like Divya who you know, they kind of helped with that that deployment story, you still you still get your Django project which isn't quite as easy to get off the ground as a WordPress you know, for instance and so it it would be really nice if there were you know, maybe on top of wagtail or on some some other way but if it were this kind of you can get a blog up and running pretty Yeah,
Will Vincent 48:40
I think that object model and react has I mean, it's really nice because you know, objecting is it's pretty, pretty forceful. Once you object you're out but you can go a long way and write I guess you can go quite quite a long, long runway and they do a pretty good job of saying, Yeah, once you eject, you're on your own.
Tom Dyson 48:58
Yeah, yeah. I've been trying I imagine from a user point of view, what the best possible way of what the best user experience would be for deploying the site. And you're right. And this is the question that most people are asking. You see this on Reddit and Stack Overflow. It's, you know, I've been to the tool, I've built my site. Now I'm stuck because I can't get nginx to talk to a unicorn.
Will Vincent 49:19
Yeah, well,
Tom Dyson 49:20
yeah. It's tricky stuff. And it's, you know, and there's lots of different edge cases and different different ways that it can fail. So my
Carlton Gibson 49:29
entire living doing just that
Tom Dyson 49:31
we not have a new manager by command manager at pi deploy.
It says, Do you want it to grow? Yes,
yes to Roku or DeVito or nginx. Or, you know,
Will Vincent 49:44
yeah, we Yeah, we should have that.
Carlton, I know you agree.
Carlton Gibson 49:50
Oh, look, I look a flying pony.
Will Vincent 49:54
Yeah, right. Well,
Carlton Gibson 49:57
yeah, no, I think Yeah, I do agree. I think that's it. Still the part of the story, which is most difficult, but
Will Vincent 50:03
I think you just have to be really opinionated with it is the thing is that you almost need to not ask a Django developer, but you know, get a group of these beginners and, you know, optimize for that. And, you know, we can talk about the trade offs all day long. But, yeah, I can act when I need to. But, you know, we sort of have the wrong perspective on it. I would say,
Tom Dyson 50:22
I think you're right. Yeah. Talk to the beginner.
Will Vincent 50:24
Yeah. I mean, you know, the problem then is that this is the problem with Heroku. And all these platforms as a service is that it's a it's a business model challenge, because then you get all the beginners and intermediate people and then once they become sites that would actually pay the bills. They're tempted to, to leave but I think even that is changing. I mean, it just makes so much sense for most sites to have a managed hosting solution. But Great, well, thank you so much for joining us, Tom. We're gonna have links to all the things we've mentioned. there anything Carlton, you want to add?
Carlton Gibson 50:56
No, super. I mean, you know, normally, I'd say more, but Tom's Hit all the points right in the head so I can just sit and listen.
Will Vincent 51:02
That's great. Is there a way for people to contact or contribute to wagtail? Or get involved with it, Tom?
Tom Dyson 51:08
Absolutely. Yep. So it's GitHub slash wagtail slash wagtail. And there's a there's a Slack channel at octo IO forward slash slack. Those are the probably the easiest entry points. Who email me Tommy talks calm and really, I'm really happy to help anyone get started. Wow, that's,
Will Vincent 51:27
that's a gutsy move, giving your email like, that's great. Well, and for this podcast, you can always see new episodes at Jango chat calm. Please leave a review and we'll see you all next week. Bye. Bye. Thanks for joining us.