As someone who worked for years in commercial print, before most manufacturing moved overseas, I recall the workflows the article discusses as being more automate-able than the author seems to understand. For example, "Making the slightest change became a chore. [1.] Update the 'master' DOCX. [2.] Update the InDesign file ..." --the appropriate way to use an external document as master in InDesign is to use the Place command, which autoupdates text changes as they are made in Word. As another example, InDesign supports multiple formats of EPUB by direct export. I also question the author's familiarity with common LaTeX workflows. "'Why didn’t you just author it in LaTeX? ...' you might ask. [B]ut I prefer writing novels in a word processor, not a text editor." And, "How do I convert an ODT file to TeX?" Word processors offer exports of all kinds, including to plain text, and the purpose of a TeX editor is, like InDesign, to typeset text that is often written elsewhere. Capturing the styling from the word processor seems antithetical to the desire for an advanced typesetting tool.
Overall, as a technical writeup I enjoyed the article; however, I would caution that the author seems to approach publishing from an amateur perspective.
The place command does not autoupdate. At least not in the most recent version.
Text is either embedded, in which case it's baked in, or linked, in which case you have to manually tell ID to update the link to reload the text.
But InDesign's EPUB output is horrifically terrible, especially if you're trying to use custom fonts/graphics for page headings. (Basically - no.)
And the CSS is... really not great.
The best fiction off-the-shelf option for EPUB gen is Vellum. It's a one-off payment of around $250 and you can get an EPUB-only version, or EPUB+PDF for print. It's not very customisable, but the presets - there aren't many - all look good.
For anything more sophisticated, options are limited. I spent far too long creating a non-fiction EPUB in ID a couple of years ago. I got there in the end but it was an extremely painful process and I ended up automating a lot of the workflow in JSX.
For fiction I created my own MD -> EPUB pipeline with a custom MD -> HTML parser for custom markup not handled by pure MD. Then a custom EPUB builder which does all the wrapping and general EPUB bureaucracy based on my own CSS.
Python has libraries for Pandoc, native DOCX, and MD (up to a point) so the basics were all there. The rest was glue.
It was a moderately-sized hobby project - would probably go much faster with AI now.
> Overall, as a technical writeup I enjoyed the article; however, I would caution that the author seems to approach publishing from an amateur perspective.
I also worked at a publishing company (for ~6 years) in the early 2000s. While you are right that the pros have some tricks to make the process easier, the fact remains that the process is not easy at all. Unlike in academic publishing, where nothing stands between the author and the reader, at a commercial publishing company (at least one of the majors), there are legions of people working behind the scenes. Editors communicate with authors; editorial assistants help the editors with fact-checking, drafts, basic organization and comprehensibility; copyeditors get all pedantic about formatting and word choice (sometimes resulting in arguments with authors that the editors need to smooth over); production departments that make the books look pretty, contain images whose copyrights are cleared and that can be legibly printed within a reasonable budget; graphic designers who develop house styles or even a custom style for a book and even original cover art; lawyers who negotiate copyrights for excerpts, images, and other ancillary materials; and on and on.
I know all this because I worked on a custom content management system for this company and in so doing I discovered that the process was incredibly complex. One of the major pet peeves of everybody involved was when an author thought they were doing anybody a favor by trying format things in Microsoft Word. Most of that information was thrown away and the real layout was done by people who thought in terms of widows, orphans, kerning, and leading (and so on). Once you know what all the people in a top publishing company do, the difference between an amateur publication and a professional one becomes immediately apparent. So I don't fault the author for getting a bit technical. The SE approach sounds like an epic attempt to make a complicated subject at least somewhat approachable.
When we get a Word doc from an author it is sent to the typesetter for reformatting. A standard set of style codes is applied and other corrections made so it can be directly imported into the design template. This the version the copyeditor works on. Also: once proofs are set this version is basically trash. In ye olde dayes, when this was all done on paper, the edited ms would eventually go back to the author, but sometimes they didn't want it. Now when the book is done the production manuscript files get deleted.
For ebook production, you could definitely do worse than follow Standard Ebooks' method. That will get you a decent standards-compliant file with basic accessibility features accounted for.
For my part, my approach was to set up a Word .docx file with styles, which would import into Adobe InDesign, mapping style-to-style, and if need be, pre-process w/ one or more AppleScripts and page as normal, then when it was time to return the edited manuscript to the author(s), select all the text and remove over-rides and export the text as a .rtf from InDesign, open that in Microsoft Word and re-save as a .docx.
I've been publishing print and ebooks since 2015, and I can attest to the fact the Word to PDF X-1/a to epub/kindle pipeline is painful. Making minor edits after publication is also painful, as the author notes, and can be error prone if you fail to make identical changes to all formats.
The problem was bad enough that I built by own markdown to HTML to PDF/X-1a processor using Python, WeasyPrint, and ghostscript. This also allows me to use git for version control, and I can make formatting changes using vanilla CSS. My tools are currently too crude for the average non-tech writer to use, but they save me hours every time I use them.
For any of you hackers out there looking for an untapped market, try making a user-friendly tool that converts Word, PDF and/or similar formats to the print-ready PDF/X-1a, PDF/X-3 and PDF/X-4 formats. At the moment, all the existing tools are proprietary and expensive, and many are difficult to use. This won't be a big money maker, but it will certainly be welcome by many indie authors.
Setting up good book publishing pipelines with version control + CI/CD might sounds simple, but I don't think it's trivial.
One of the best examples of this that I've ever seen is The Sourdough Framework [0] -- really impressed with the way that versioning and publishing is integrated in that book.
And yes -- I know it sounds like yet another Javascript library -- but it's actually a book about sourdough bread making. It's been discussed here several times before, but this one from 2023 [1] may have been the most popular (103 comments)
Uhm why not Typst? I published my thesis and another book in it and it worked great. They are also working on HTML output which should make it easier to create EPUBs. Until then Pandoc should work I think
My only problem using git and a text editor is deciding whether I want hard or soft wraps. Vim handles hard wraps better IMO and you can change the git diff engine to something like difft, which makes it much more bearable than the default for hard wrap prose.
But softwrap definitely has its advantages: no hard line breaks makes copying the text into other mediums easier, git diffs show only which paragraphs you edited and not a bunch of line diff noise no matter which engine you use. Only problem is it breaks my yy, dd, cc muscle memory, as AFAIK you can't force those to work on virtual (vs logical) lines.
The annoyances of using "soft wraps" with various kinds of tools is one of the maddening irritations of our software landscape. Inserting non-semantic newlines in content just to make things fit the screen is insane.
Did the author create the Christian novellas he's mentioned? Can't tell by the phrasing. That would be impressive enough on its own, combined with the tech stack?
D. J. Speckhals is the author of the “Witnesses of the Light” historical fiction trilogy, which transports readers to fifteenth-century Europe to explore the resilient faith of the Waldensians.
I love buying and reading physical books. However, about half of the books (I read mostly programming books) have letters that are printed pixelated. This is infuriating to me. No one bothers to run a trial print and see what comes out?
The root cause of this: PDF will look fine, but the text color is usually set slightly off black (why!!??). The eye couldn’t really see the difference and PDF renders smoothly. However, commercial printers couldn’t handle that properly.
Solution: set the text color to full black, you are using (most of the time) black and white printer!
You might need to have two PDF versions: one for printing and one for digital distribution (but why would you have off-black text anyway?).
Because pure black causes eye strain. Dark gray on white is superior for long reading sessions when your paper is white. The contrast really hurts after a while if you do pure black on pure white. This is a known phenomenon.
In fact, there's experimental evidence (https://www.nature.com/articles/s41598-018-28904-x) that this high contrast plays a hand in the onset of myopia, which in extreme forms is correlated with glaucoma and other vision disorders.
AKA what CS PhD students have been doing ~forever.
I guess this is like medical researchers "discovering" basic calculus or an office worker discovering that SFTP, sshfs, and git work fine and they don't need Dropbox after all.
What's common knowledge in one field can apparently still be alien to people outside the field, even in the age of LLMs.
Just wait until the author finds out about Overleaf...
As someone who worked for years in commercial print, before most manufacturing moved overseas, I recall the workflows the article discusses as being more automate-able than the author seems to understand. For example, "Making the slightest change became a chore. [1.] Update the 'master' DOCX. [2.] Update the InDesign file ..." --the appropriate way to use an external document as master in InDesign is to use the Place command, which autoupdates text changes as they are made in Word. As another example, InDesign supports multiple formats of EPUB by direct export. I also question the author's familiarity with common LaTeX workflows. "'Why didn’t you just author it in LaTeX? ...' you might ask. [B]ut I prefer writing novels in a word processor, not a text editor." And, "How do I convert an ODT file to TeX?" Word processors offer exports of all kinds, including to plain text, and the purpose of a TeX editor is, like InDesign, to typeset text that is often written elsewhere. Capturing the styling from the word processor seems antithetical to the desire for an advanced typesetting tool.
Overall, as a technical writeup I enjoyed the article; however, I would caution that the author seems to approach publishing from an amateur perspective.
The place command does not autoupdate. At least not in the most recent version.
Text is either embedded, in which case it's baked in, or linked, in which case you have to manually tell ID to update the link to reload the text.
But InDesign's EPUB output is horrifically terrible, especially if you're trying to use custom fonts/graphics for page headings. (Basically - no.)
And the CSS is... really not great.
The best fiction off-the-shelf option for EPUB gen is Vellum. It's a one-off payment of around $250 and you can get an EPUB-only version, or EPUB+PDF for print. It's not very customisable, but the presets - there aren't many - all look good.
For anything more sophisticated, options are limited. I spent far too long creating a non-fiction EPUB in ID a couple of years ago. I got there in the end but it was an extremely painful process and I ended up automating a lot of the workflow in JSX.
For fiction I created my own MD -> EPUB pipeline with a custom MD -> HTML parser for custom markup not handled by pure MD. Then a custom EPUB builder which does all the wrapping and general EPUB bureaucracy based on my own CSS.
Python has libraries for Pandoc, native DOCX, and MD (up to a point) so the basics were all there. The rest was glue.
It was a moderately-sized hobby project - would probably go much faster with AI now.
> The rest was glue
Oh how often I keep saying that these days... "All the parts are there! Why hasn't anyone piped this into that?"
> Overall, as a technical writeup I enjoyed the article; however, I would caution that the author seems to approach publishing from an amateur perspective.
I also worked at a publishing company (for ~6 years) in the early 2000s. While you are right that the pros have some tricks to make the process easier, the fact remains that the process is not easy at all. Unlike in academic publishing, where nothing stands between the author and the reader, at a commercial publishing company (at least one of the majors), there are legions of people working behind the scenes. Editors communicate with authors; editorial assistants help the editors with fact-checking, drafts, basic organization and comprehensibility; copyeditors get all pedantic about formatting and word choice (sometimes resulting in arguments with authors that the editors need to smooth over); production departments that make the books look pretty, contain images whose copyrights are cleared and that can be legibly printed within a reasonable budget; graphic designers who develop house styles or even a custom style for a book and even original cover art; lawyers who negotiate copyrights for excerpts, images, and other ancillary materials; and on and on.
I know all this because I worked on a custom content management system for this company and in so doing I discovered that the process was incredibly complex. One of the major pet peeves of everybody involved was when an author thought they were doing anybody a favor by trying format things in Microsoft Word. Most of that information was thrown away and the real layout was done by people who thought in terms of widows, orphans, kerning, and leading (and so on). Once you know what all the people in a top publishing company do, the difference between an amateur publication and a professional one becomes immediately apparent. So I don't fault the author for getting a bit technical. The SE approach sounds like an epic attempt to make a complicated subject at least somewhat approachable.
[delayed]
When we get a Word doc from an author it is sent to the typesetter for reformatting. A standard set of style codes is applied and other corrections made so it can be directly imported into the design template. This the version the copyeditor works on. Also: once proofs are set this version is basically trash. In ye olde dayes, when this was all done on paper, the edited ms would eventually go back to the author, but sometimes they didn't want it. Now when the book is done the production manuscript files get deleted.
For ebook production, you could definitely do worse than follow Standard Ebooks' method. That will get you a decent standards-compliant file with basic accessibility features accounted for.
For my part, my approach was to set up a Word .docx file with styles, which would import into Adobe InDesign, mapping style-to-style, and if need be, pre-process w/ one or more AppleScripts and page as normal, then when it was time to return the edited manuscript to the author(s), select all the text and remove over-rides and export the text as a .rtf from InDesign, open that in Microsoft Word and re-save as a .docx.
Kudos to you for doing that.
I've been publishing print and ebooks since 2015, and I can attest to the fact the Word to PDF X-1/a to epub/kindle pipeline is painful. Making minor edits after publication is also painful, as the author notes, and can be error prone if you fail to make identical changes to all formats.
The problem was bad enough that I built by own markdown to HTML to PDF/X-1a processor using Python, WeasyPrint, and ghostscript. This also allows me to use git for version control, and I can make formatting changes using vanilla CSS. My tools are currently too crude for the average non-tech writer to use, but they save me hours every time I use them.
For any of you hackers out there looking for an untapped market, try making a user-friendly tool that converts Word, PDF and/or similar formats to the print-ready PDF/X-1a, PDF/X-3 and PDF/X-4 formats. At the moment, all the existing tools are proprietary and expensive, and many are difficult to use. This won't be a big money maker, but it will certainly be welcome by many indie authors.
Setting up good book publishing pipelines with version control + CI/CD might sounds simple, but I don't think it's trivial.
One of the best examples of this that I've ever seen is The Sourdough Framework [0] -- really impressed with the way that versioning and publishing is integrated in that book.
And yes -- I know it sounds like yet another Javascript library -- but it's actually a book about sourdough bread making. It's been discussed here several times before, but this one from 2023 [1] may have been the most popular (103 comments)
[0] - https://github.com/hendricius/the-sourdough-framework [1] - https://news.ycombinator.com/item?id=35961590
Sad that typst wasn’t mentioned, wonder how it compares to the setup in the article.
Uhm why not Typst? I published my thesis and another book in it and it worked great. They are also working on HTML output which should make it easier to create EPUBs. Until then Pandoc should work I think
My only problem using git and a text editor is deciding whether I want hard or soft wraps. Vim handles hard wraps better IMO and you can change the git diff engine to something like difft, which makes it much more bearable than the default for hard wrap prose.
But softwrap definitely has its advantages: no hard line breaks makes copying the text into other mediums easier, git diffs show only which paragraphs you edited and not a bunch of line diff noise no matter which engine you use. Only problem is it breaks my yy, dd, cc muscle memory, as AFAIK you can't force those to work on virtual (vs logical) lines.
The annoyances of using "soft wraps" with various kinds of tools is one of the maddening irritations of our software landscape. Inserting non-semantic newlines in content just to make things fit the screen is insane.
Did the author create the Christian novellas he's mentioned? Can't tell by the phrasing. That would be impressive enough on its own, combined with the tech stack?
From the about page:
Hopefully some of the writers are reading this:
I love buying and reading physical books. However, about half of the books (I read mostly programming books) have letters that are printed pixelated. This is infuriating to me. No one bothers to run a trial print and see what comes out?
The root cause of this: PDF will look fine, but the text color is usually set slightly off black (why!!??). The eye couldn’t really see the difference and PDF renders smoothly. However, commercial printers couldn’t handle that properly.
Solution: set the text color to full black, you are using (most of the time) black and white printer!
You might need to have two PDF versions: one for printing and one for digital distribution (but why would you have off-black text anyway?).
This is why PDF/X exists
> why is the text color set slightly off black
Because pure black causes eye strain. Dark gray on white is superior for long reading sessions when your paper is white. The contrast really hurts after a while if you do pure black on pure white. This is a known phenomenon.
In fact, there's experimental evidence (https://www.nature.com/articles/s41598-018-28904-x) that this high contrast plays a hand in the onset of myopia, which in extreme forms is correlated with glaucoma and other vision disorders.
AKA what CS PhD students have been doing ~forever.
I guess this is like medical researchers "discovering" basic calculus or an office worker discovering that SFTP, sshfs, and git work fine and they don't need Dropbox after all.
What's common knowledge in one field can apparently still be alien to people outside the field, even in the age of LLMs.
Just wait until the author finds out about Overleaf...
> what CS PhD students have been doing ~forever.
Or what every researcher has been doing for literally decades (except with other versioning systems, but still typesetting without Word or Adobe).
No need for techbros to pat themselves on the back as innovators.
I typeset my novels in LaTeX and use GIT. I even just clone a base repo whenever I'm going to release another.