Re: [LAD] Non Session Management

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Emanuel Rumpf <xbran@...>
Cc: <Linux-audio-dev@...>
Date: Tuesday, March 27, 2012 - 1:23 am

On Tue, 2012-03-27 at 01:04 +0200, Emanuel Rumpf wrote:

This exact problem came up when I set out to implement non-destructive
plugin state saving in Ardour, with support for plugins referring to
files (e.g. loaded samples).

The obvious basic solution in a non-destructive context is to save each
snapshot to its own directory. The tricky bits come when you have links
in that directory to files. What you end up with is a few directories
with specific roles.

I'll try to document the path leading to this as tersely as possible.
The plugin/host situation is analogous to app/SM, very closely with
respect to what's on disk, not so much with respect to responsibilities
and such.

There are two kinds of file:

* External files, e.g. something you loaded from the filesystem
somewhere. There are assumed to be immutable.

* Files created by the plugin, which may be different each save

As I have said a few times on this list, for various reasons I think
straightforward and transparent symlinks are the best way to refer to
external files. I'll just take this as given/obvious and not bother
rehashing the argument (erecting a bunch of protocol and/or file format
junk only *adds* problems).

So, you want to save state, and the plugin refers to a bunch of files.
As mentioned above, this can mean a lot of references to one file.
Since these could be massive, redundant copies must never happen at
all. For ease of moving things, or resolving them if they break, we
want a minimal number of actual links to an external file, i.e. 1 (this
is also necessary for archival, see below). So, we don't want every
single state snapshot directory to have a link to /media/bigfile.wav.
To avoid this, make a directory specifically for all links to external
file, and link to those links instead. This localizes all links to
files outside the session.

The other case is files written to during execution, e.g. recorded
waveforms. Here, on each save, you need to check if the file is
actually any different, and make an actual copy if so (so the original
can continue to be used) in the state snapshot directory. During run
time, all the created files live in a "scratch" directory, which you can
preserve, or just throw out when you close since copies have been made
for every snapshot.

Working with the requirement that the plugin's file namespace must not
be messed with (e.g. if you save foo.wav to your state directory, it
should actually be called foo.wav), you end up with:

1) The "file directory". This is where the plugin creates files,
whenever. It can be considered run-time scratch.

2) The "copy directory". This is where copies of things in the file
directory are made, to preserve their state at a particular instant in
time. This mirrors the structure of the file directory, but appends .2
or .3 etc. for the various snapshots of a file

3) The "save directory". This is the directory of a particular state
snapshot. It can contain whatever.

4) The "link directory". This is where all links to external resources
live. Any state snapshot that refers to an external file actually links
to a link here.

This is in the Lilv API. It might seem complicated, but it is only
optionally this powerful, if you don't care about non-destructive saving
or avoiding copies you can simply use one directory for everything.

I think I can make a reasonable argument that this is the minimal
number of directories that meet the desired requirements:

A) The file directory and copy directory can not be the same directory,
because the copies would pollute the plugin's file namespace and
possibly clash. It would also make it difficult to know what files are
actually needed by the session, since some may be scratch. The files
directory is unique in that its contents may be safely deleted.

B) The copy directory and save directory can not be the same directory,
since multiple saves may refer to the same file with the same state.
Without the copy directory this would require two copies of the same
file.

C) The link directory must be separate to avoid having a large number
of links to the same external file. This is convenient in general
(only one link to potentially fix), but required for archival. An
archival tool (e.g. tar -h) would actually copy the file to the links
location. Without the link directory this would produce an archived
session with many copies of the same (possibly massive) file.

D) The save directory is inherently where one save takes place, its
existence is a given.

Figuring this all out was a long process of trial and error, but now
that I write it down it seems clear it can't be simplified without
failing to meet a requirement. I am all ears for arguments to the
contrary, though.

This scheme would need some slight tinkering to tolerate external files
that change, but this would require keeping copies of system files
around just in case, not just when you archive. That is, ANY external
files used EVER would have to be copied in case they change. I don't
think this is acceptable, and plugins have no business modifying
external files anyway. Apps that want to get fancy and share files may
want a solution to this, but particularly since shared mutable files is
almost certainly crazy anyway, one app should simply "own" any such
files. It may tell other hosts about them if it pleases, possibly via
OSC. I don't think the session manager itself should be bloated out
with a bunch of unnecessarily shared file database stuff to support a
rather esoteric ability that can easily be achieved without doing so.
In other words, if apps want to talk to each other and share file paths,
that is their business.

-dr

_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
[LAD] Non Session Management, rosea.grammostola, (Thu Mar 22, 10:55 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Thu Mar 22, 4:52 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 22, 6:59 pm)
Re: [LAD] Non Session Management, David Robillard, (Thu Mar 22, 3:10 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Sat Mar 24, 3:19 pm)
Re: [LAD] Non Session Management, Fons Adriaensen, (Sat Mar 24, 10:09 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 9:24 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Thu Mar 29, 1:18 pm)
Re: [LAD] Non Session Management, Louigi Verona, (Thu Mar 29, 10:02 am)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 10:41 am)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 11:08 am)
Re: [LAD] Non Session Management, thijs van severen, (Thu Mar 29, 10:30 am)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 10:46 am)
Re: [LAD] Non Session Management, thijs van severen, (Thu Mar 29, 11:17 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Thu Mar 29, 2:22 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 2:35 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 11:21 am)
Re: [LAD] Non Session Management, thijs van severen, (Thu Mar 29, 11:26 am)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 29, 11:34 am)
Re: [LAD] Non Session Management, rosea.grammostola, (Mon Mar 26, 3:42 pm)
Re: [LAD] Non Session Management, Louigi Verona, (Mon Mar 26, 3:51 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Mon Mar 26, 4:16 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Mon Mar 26, 6:15 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Mon Mar 26, 8:32 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Mon Mar 26, 9:47 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Wed Mar 28, 7:25 pm)
Re: [LAD] Non Session Management, J. Liles, (Wed Mar 28, 8:43 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Wed Mar 28, 9:20 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Wed Mar 28, 9:28 pm)
Re: [LAD] Non Session Management, Fons Adriaensen, (Mon Mar 26, 9:40 pm)
Re: [LAD] Non Session Management, J. Liles, (Tue Mar 27, 12:15 am)
Re: [LAD] Non Session Management, Fons Adriaensen, (Tue Mar 27, 7:41 pm)
Re: [LAD] Non Session Management, Lieven Moors, (Wed Mar 28, 9:28 am)
Re: [LAD] Non Session Management, Fons Adriaensen, (Wed Mar 28, 8:19 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Tue Mar 27, 8:45 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Wed Mar 28, 1:28 am)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 3:46 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Wed Mar 28, 12:24 pm)
Re: [LAD] Non Session Management, Burkhard W├Âlfel, (Fri Mar 30, 2:46 pm)
Re: [LAD] Non Session Management, David Robillard, (Fri Mar 30, 5:41 pm)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 5:26 pm)
Re: [LAD] Non Session Management, thijs van severen, (Wed Mar 28, 1:36 pm)
Re: [LAD] Non Session Management, Renato, (Wed Mar 28, 4:35 pm)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 5:33 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Wed Mar 28, 6:01 pm)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 6:53 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Thu Mar 29, 1:30 pm)
Re: [LAD] Non Session Management, David Robillard, (Thu Mar 29, 6:21 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Mon Apr 2, 9:33 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Wed Mar 28, 12:12 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Mon Mar 26, 11:05 pm)
Re: [LAD] Non Session Management, David Robillard, (Tue Mar 27, 1:23 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Tue Mar 27, 12:24 pm)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 3:43 am)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Wed Mar 28, 12:11 pm)
Re: [LAD] Non Session Management, David Robillard, (Wed Mar 28, 5:01 pm)
Re: [LAD] Non Session Management, David Robillard, (Mon Mar 26, 9:59 pm)
Re: [LAD] Non Session Management, J. Liles, (Mon Mar 26, 7:04 pm)
Re: [LAD] Non Session Management, Louigi Verona, (Mon Mar 26, 4:21 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Mon Mar 26, 4:30 pm)
Re: [LAD] Non Session Management, Diego Simak, (Mon Mar 26, 8:17 pm)
Re: [LAD] Non Session Management, Emanuel Rumpf, (Mon Mar 26, 8:42 pm)
Re: [LAD] Non Session Management, Diego Simak, (Mon Mar 26, 9:39 pm)
Re: [LAD] Non Session Management, thijs van severen, (Thu Mar 22, 11:59 am)
Re: [LAD] Non Session Management, David Robillard, (Thu Mar 22, 2:54 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 22, 12:33 pm)
Re: [LAD] Non Session Management, Harry van Haaren, (Thu Mar 22, 1:52 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 22, 3:43 pm)
Re: [LAD] Non Session Management, Tim E. Real, (Sun Mar 25, 7:35 am)
Re: [LAD] Non Session Management, Fons Adriaensen, (Sun Mar 25, 10:44 am)
Re: [LAD] Non Session Management, thijs van severen, (Thu Mar 22, 12:57 pm)
Re: [LAD] Non Session Management, rosea.grammostola, (Thu Mar 22, 12:53 pm)