The original post asked for an example:
ThereThe Babington Plot is a good story of a bad cryptosystem causing problems. Mary Queen of Scots was imprisoned by her cousin Queen Elizabeth I and was communicating with people on the outside via encrypted letters. The alphabet was replaced with a cryptoalphabet of squiggles, crossed circles and triangles with extra letters assigned for common letters, like e,t,i and o so the meaning of letters could not be found quickly by frequency analysis. They also added a few null characters which were ignored on the decryption to throw off analysts. The problem was that The Queen had a very competent cryptanalyst on her staff in the person of Thomas Phelippes who was able to decrypt the messages as they were intercepted.
As things progressed, Mary went along with a plot to have her escape and take over the throne. When The Queen’s agents intercepted the last letter from Mary before passing it on, they added an encrypted sentence asking for the names of those involved in the plot “so they can be properly rewarded.” Mary’s correspondent dutifully answered and The Queen’s agents had everybody involved executed.
When my kids were small, I would send them cryptograms with their lunch (with a key (using a Vernam cipher)). Generally, they were jokes but they were never of any importance. In a case like that, roll your own is fine. If you are plotting to overthrow The Queen of England (or the Shah of Iran or the slowly reforming Thug-ocracy in Myanmar), I would suggest that you make sure that what you are using cannot be easily decrypted. I think it was Bruce Schneier that saidAs Bruce Schneier said, anybody can come up with a cryptosystem they cannot decrypt, but coming up with one nobody else can decrypt is harder.