Commit Graph

350 Commits

Author SHA1 Message Date
Jaime Marquínez Ferrándiz 360e1ca5cc [youtube] Convert to new subtitles system
The automatic captions are stored in the 'automactic_captions' field, which is used if no normal subtitles are found for an specific language.
2015-02-16 22:47:39 +01:00
Jaime Marquínez Ferrándiz c84dd8a90d [YoutubeDL] store the subtitles to download in the 'requested_subtitles' field
We need to keep the orginal subtitles information, so that the '--load-info' option can be used to list or select the subtitles again.
We'll also be able to have a separate field for storing the automatic captions info.
2015-02-16 21:51:08 +01:00
Jaime Marquínez Ferrándiz a504ced097 Improve subtitles support
For each language the extractor builds a list with the available formats sorted (like for video formats), then YoutubeDL selects one of them using the '--sub-format' option which now allows giving the format preferences (for example 'ass/srt/best').
For each format the 'url' field can be set so that we only download the contents if needed, or if the contents needs to be processed (like in crunchyroll) the 'data' field can be used.

The reasons for this change are:
* We weren't checking that the format given with '--sub-format' was available, checking it in each extractor would be repetitive.
* It allows to easily support giving a format preference.
* The subtitles were automatically downloaded in the extractor, but I think that if you use for example the '--dump-json' option you want to finish as fast as possible.

Currently only the ted extractor has been updated, but the old system still works.
2015-02-16 21:51:03 +01:00
Philipp Hagemeister 73fac4e911 [ffmpeg] Add --ffmpeg-location 2015-02-16 04:05:53 +01:00
Philipp Hagemeister 442c37b7a9 [YoutubeDL] Do not perform filter matching on partial results (Fixes #4921) 2015-02-10 11:28:28 +01:00
Philipp Hagemeister 7e5db8c930 [options] Add --no-color 2015-02-10 04:22:10 +01:00
Philipp Hagemeister 347de4931c [YoutubeDL] Add generic video filtering (Fixes #4916)
This functionality is intended to eventually encompass the current format filtering.
2015-02-10 03:32:24 +01:00
Philipp Hagemeister 43f244b6d5 [YoutubeDL] Do not show worst in --list-formats output
Nobody wants to know what the worst possible format is. And if they do, they can still provide -f worst.
2015-02-09 15:57:42 +01:00
Sergey M․ 2ec19e9558 [YoutubeDL] Allow filtering by audio sampling rate 2015-02-09 01:09:45 +06:00
Sergey M․ 9ddb6925bf [YoutubeDL] Allow filtering by string properties (#4906) 2015-02-09 01:07:43 +06:00
Jaime Marquínez Ferrándiz b3a286d69d [YoutubeDL] _calc_cookies: add get_header method to _PseudoRequest (#4861) 2015-02-06 22:23:06 +01:00
Sergey M․ 300753a069 [YoutubeDL] Fix video+audio format field (Closes #4880) 2015-02-06 00:51:16 +06:00
Philipp Hagemeister f6c24009be [YoutubeDL] Calculate thumbnail IDs automatically 2015-02-03 10:52:22 +01:00
Philipp Hagemeister 8f9312c387 Appease pyflakes8-3 2015-02-01 11:30:56 +01:00
Sergey M․ 6d593c3276 [YoutubeDL] Fix video+audio format_id (Closes #4824) 2015-01-31 03:50:11 +06:00
codesparkle 5468ff4d91 Remove duplicate dictionary keys 2015-01-30 20:11:51 +11:00
Jaime Marquínez Ferrándiz a7a14d9586 [YoutubeDL] set the 'thumbnails' field if the info_dict has the 'thumbnails' field
Since the '--write-thumbnail' uses the 'thumbnails' field and we didn't updated the info_dict, it wouldn't detect the thumbnail. (fixes #4812)
2015-01-29 20:15:38 +01:00
Sergey M․ dcf53d4408 [YoutubeDL] Set format_id for video+audio (Closes #3634) 2015-01-29 00:44:40 +06:00
Sergey M․ 3a0d2f520a [YoutubeDL] Temporary fix for subprocess encoding issues on python2 @ Windows (Closes #4787)
For now filenames will be encoded with preferrefencoding before written to disk
2015-01-27 22:38:28 +06:00
Philipp Hagemeister 9f0df77ab1 [YoutubeDL] Allow format filtering by fps 2015-01-26 14:36:22 +01:00
Jaime Marquínez Ferrándiz e72c7e4123 [YoutubeDL] Always set the '_filename' field in the info_dict (reported in #4053)
It's also useful when you use the '--write-info-json' option.
2015-01-26 12:01:43 +01:00
Philipp Hagemeister ee69b99af6 [YoutubeDL] clarify hook documentation 2015-01-25 06:15:54 +01:00
Philipp Hagemeister 881e6a1f5c Add --xattr-set-filesize option (Fixes #1348) 2015-01-25 04:49:44 +01:00
Philipp Hagemeister c14e88f0f5 [YoutubeDL] Add --playlist-items option (Fixes #2662) 2015-01-25 04:24:55 +01:00
Philipp Hagemeister ec82d85acd [YoutubeDL] Implement --write-all-thumbnails (Closes #2269) 2015-01-25 03:11:12 +01:00
Philipp Hagemeister cfb56d1af3 Add --list-thumbnails 2015-01-25 02:43:19 +01:00
Jaime Marquínez Ferrándiz 1070711d60 [YoutubeDL._calc_cookies] Restore the 'is_unverifiable'
I should have check everything was copied before commiting 4b405cfc6e.
2015-01-24 20:12:47 +01:00
Jaime Marquínez Ferrándiz 4b405cfc6e [YoutubeDL._calc_cookies] Restore the 'has_header' method
I didn't copied it from downloader/external
2015-01-24 20:08:24 +01:00
Jaime Marquínez Ferrándiz e5660ee6ae [YoutubeDL] Fill the info dict 'http_headers' field with all the headers available
Useful for external tools using the json output.

The methods '_calc_headers' and '_calc_cookies' have been copied from the downloader/external, now they just use "info_dict['http_headers']".
2015-01-24 18:56:04 +01:00
Philipp Hagemeister 222516d97d [downloader] Lay groundwork for external downloaders.
This comes with a very simply implementation for wget; the real work is in setting up the infrastructure.
2015-01-24 01:38:48 +01:00
Philipp Hagemeister a055469faf [downloader] Improve downloader selection 2015-01-23 23:50:31 +01:00
Philipp Hagemeister 62cd676c74 [youtube] Fixup DASH m4a headers
This fixes #2288, #2506, #2607, #3681, #4741, #4767.
2015-01-23 18:39:12 +01:00
Philipp Hagemeister 5f0d813d93 Merge remote-tracking branch 'rupertbaxter2/master'
Conflicts:
	youtube_dl/__init__.py
	youtube_dl/downloader/common.py
2015-01-23 12:05:01 +01:00
Philipp Hagemeister 083c9df93b [YoutubeDL] Allow filtering by properties (Fixes #4584) 2015-01-23 00:04:05 +01:00
Pierre 4340decad2 check for overwriting files in the downloader (fixes #3916, closes #3829) 2015-01-11 12:02:27 +01:00
Jaime Marquínez Ferrándiz f3ff1a3696 YoutubeDL: Make the decision about removing the original file after each postprocessor is run (fixes #2261)
If one of the processors said the file should be kept, it wouldn't pay
attention to the response from the following processors. This was wrong if the
'keep_video' option was False, if the first extractor modifies the original file
and then we extract its audio we don't want to keep the original video file.
2015-01-11 11:35:18 +01:00
Philipp Hagemeister 8bfa75451b [options] Add --no-call-home
While we're at it, also drop "experimental" moniker for --call-home - should work fine.
2015-01-10 21:09:18 +01:00
Philipp Hagemeister 58b1f00d19 [YoutubeDL] Add new --call-home option for debugging 2015-01-10 21:02:27 +01:00
Philipp Hagemeister be4a824d74 Add new option --source-address
Closes #3618, fixes #721, fixes #2481, fixes #4551, closes #1020.
2015-01-10 19:56:51 +01:00
Philipp Hagemeister 6271f1cad9 [youtube|ffmpeg] Automatically correct video with non-square pixels (Fixes #4674) 2015-01-10 05:45:51 +01:00
Philipp Hagemeister 45598f1578 Merge remote-tracking branch 'aft90/merge-output-format'
Conflicts:
	youtube_dl/YoutubeDL.py
2015-01-10 01:59:14 +01:00
Andrei Troie d02115f837 Use the option in preparing the merge output filename 2015-01-10 00:29:06 +00:00
Philipp Hagemeister f90ad27375 [YoutubeDL] Copy over format metadata when merging (Fixes #4671) 2015-01-09 20:50:23 +01:00
Philipp Hagemeister 0217c78377 [YoutubeDL] Allow selection by more extensions 2015-01-08 16:14:50 +01:00
Philipp Hagemeister 0590062925 Respect age_limit when listing extractors (Fixes #4653) 2015-01-07 07:20:20 +01:00
Philipp Hagemeister e65566a9cc [youtube] Correct handling when DASH manifest is not necessary to find all formats 2015-01-03 18:33:38 +01:00
Jaime Marquínez Ferrándiz f00fd51dae Don't write the description file if info_dict['description'] is None (#3166) 2014-12-21 20:49:14 +01:00
Jaime Marquínez Ferrándiz 0e4cb4f406 YoutubeDL: style fix 2014-12-17 14:55:27 +01:00
AndroKev cd58dc3e56 Update YoutubeDL.py 2014-12-17 13:21:22 +01:00
AndroKev c33bcf2051 only add video-id to archive, when successful
Example:
no space left--> youtube-dl adds the id to archive, but the video isn't fully downloaded
2014-12-17 13:05:19 +01:00
Philipp Hagemeister 734ea11e3c Drop hash character in downloader output (#4484) 2014-12-16 00:37:42 +01:00
Philipp Hagemeister d86007873e [YoutubeDL] Document where details for format can be found 2014-12-16 00:22:23 +01:00
Philipp Hagemeister 71b640cc5b [YoutubeDL] Add declarative version of progress hooks 2014-12-15 01:26:20 +01:00
Philipp Hagemeister 4f026fafbc [YoutubeDL] Make postprocessors declarative
Instead of having to configure PPs in code, this allows us and embedding programs not to worry about imports or finer details, similarly to how we handle IEs.
2014-12-15 01:06:25 +01:00
Mark Schreiber ff815fe65a Download playlist items in reverse order
Series of videos are typically uploaded to YouTube playlists in
chronological order.  By default, these videos are downloaded
latest-to-earliest; this is great for seeing the latest videos in a
series, but prevents streaming video in the order that the videos were
produced.  Add an option to download videos in reverse order,
earliest-to-latest.

Conflicts:
	youtube_dl/YoutubeDL.py
	youtube_dl/__init__.py
2014-12-12 16:56:29 +01:00
Philipp Hagemeister 412c617d0f [cnet] Update to new theplatform infrastructure (Fixes #2736) 2014-12-12 15:55:55 +01:00
Philipp Hagemeister f17e4c9c28 [screenwavemedia] Simplify (#3766) 2014-12-12 02:11:58 +01:00
cryptonaut 16ae61f655 Handle --get-url with merged formats (fixes #2883)
Outputs one URL per line
2014-12-06 12:55:07 -08:00
Philipp Hagemeister b82f815f37 Allow iterators for playlist result entries 2014-12-06 14:02:19 +01:00
Sergey M․ 706d7d4ee7 [YoutubeDL] Avoid negative timestamps on Windows 2014-12-02 21:18:07 +06:00
Philipp Hagemeister 4aae94f9d0 [YoutubeDL] Remove incorrect documentation 2014-11-26 11:25:43 +01:00
Philipp Hagemeister acda92f6bc Clarify --no-playlist documentation (Closes #4309) 2014-11-26 10:51:03 +01:00
Jouke Waleson 9e1a5b8455 PEP8: applied even more rules 2014-11-23 21:39:15 +01:00
Jouke Waleson 8bcc875676 PEP8: more applied 2014-11-23 21:20:46 +01:00
Jouke Waleson 5f6a1245ff PEP8 applied 2014-11-23 20:41:03 +01:00
Philipp Hagemeister 7d4111ed14 Provide guidance when called with a YouTube ID starting with a dash.
Reported at https://news.ycombinator.com/item?id=8648121
2014-11-23 10:51:09 +01:00
Philipp Hagemeister 42e12102a9 [YoutubeDL] Fix multi_video check 2014-11-21 22:39:57 +01:00
Philipp Hagemeister dfd5313afd [YoutubeDL] Support new _type multi_video 2014-11-21 00:25:46 +01:00
Philipp Hagemeister c9bf41145f [YoutubeDL] Warn if an extractor returns compat_list 2014-11-20 16:29:31 +01:00
Jaime Marquínez Ferrándiz 92120217eb [cache] Fix writing to paths with unicode characters
* Use "compat_getenv"
* "write_json_file" now expects the filename to be a string
2014-11-19 00:02:24 +01:00
Philipp Hagemeister c6afed48ff [YoutubeDL] guard against strange sys.stdouts 2014-11-12 15:30:26 +01:00
Jaime Marquínez Ferrándiz c295490830 [YoutubeDL] Fix bug in the detection of formats that don't contain video (fixes #4150)
If the format requested was not available, we called the method '.get' in None.
2014-11-12 09:42:35 +01:00
Philipp Hagemeister a1cf99d03a [YoutubeDL] Add playlist_id and playlist_title fields (Fixes #4139) 2014-11-09 22:32:35 +01:00
Jaime Marquínez Ferrándiz f866e474f3 [YoutubeDL] Don't dowload formats for merging if the first doesn't contain the video (#4132) 2014-11-09 10:59:56 +01:00
Philipp Hagemeister 8c25f81bee [util] Move compatibility functions out of util
utils is large enough without these compatibility functions.

Everything that is present in newer versions of Python (i.e. with dev Python it's just an import) goes into compat.py .
Everything else (i.e. youtube-dl-specific helpers) goes into utils.py .
2014-11-02 11:23:42 +01:00
Philipp Hagemeister 4c83c96795 [YoutubeDL] Include rtmpdump in exe versions -v output 2014-11-02 10:55:36 +01:00
Philipp Hagemeister fbb21cf528 [youtube] Add formats 298, 299 (Fixes #4056) 2014-10-30 09:34:13 +01:00
Philipp Hagemeister 3511266bc3 [YoutubeDL] Simplify API of YoutubeDL
Calling add_default_extractors twice should be harmless since the first set of extractors will match.
2014-10-28 12:54:29 +01:00
Philipp Hagemeister 488447455d [ffmpeg] Warn if ffmpeg/avconv version is too old (Fixes #4026) 2014-10-26 16:46:34 +01:00
Philipp Hagemeister d28b517154 [YoutubeDL] Output avconv/ffmpeg versions if -v is given 2014-10-26 16:31:52 +01:00
Sergey M․ f889cea109 Merge branch 'compat-getenv-and-expanduser' of https://github.com/dstftw/youtube-dl into dstftw-compat-getenv-and-expanduser
Conflicts:
	test/test_utils.py
	youtube_dl/__init__.py
2014-10-26 19:56:52 +07:00
Philipp Hagemeister 63e0be3415 New option --dump-single-json (#4003) 2014-10-25 00:30:57 +02:00
Philipp Hagemeister e82c1e9a6e [YoutubeDL] Do not apply playlist info to videos when extract_flat is set (#4003) 2014-10-24 16:13:45 +02:00
Philipp Hagemeister 057a5206cc Add --flat-playlist option (Closes #4003) 2014-10-24 14:48:12 +02:00
George Boyle 53d9009bdb KeyError on initialising YoutubeDL in python3 #3910 2014-10-10 10:03:24 +01:00
George Boyle 1b725173a5 Fixed typo 2014-10-10 09:35:41 +01:00
Sergey M․ 4644ac5527 [core] Decode environment variables with filesystem encoding (Fixes #3854, Fixes #3217, Fixes #2918)
Introduces compat versions of os.getenv and os.path.expanduser
2014-09-30 22:27:53 +07:00
Sergey M․ ee0d90707a [YoutubeDL] Fix string check for python3 2014-09-28 02:48:41 +07:00
Sergey M․ 68b0973046 [YoutubeDL] Expect all kind of strings in urlopen
Now it doesn't fail if req is python2's str
2014-09-28 02:07:42 +07:00
Philipp Hagemeister 1de33fafd9 [YoutubeDL] Allow downloading multiple formats with , 2014-09-18 18:43:49 +02:00
Philipp Hagemeister e2e5dae64d Add -f m4a 2014-09-18 18:40:19 +02:00
Sergey M․ d05cfe0600 [YoutubeDL/utils] Clarify rationale for URL escaping in comment, move escape routines to utils and add some tests 2014-09-13 20:59:16 +07:00
Sergey M․ 37419b4f99 [YoutubeDL] Escape non-ASCII characters in URLs
urllib chokes on URLs with non-ASCII characters (see http://bugs.python.org/issue3991)
Working around by replacing request's original URL with escaped one
2014-09-12 23:20:17 +07:00
Philipp Hagemeister a0e07d3161 [youtube] Move cache into its own module 2014-09-03 17:29:19 +02:00
Philipp Hagemeister 241f7a8ade Merge remote-tracking branch 'JGjorgji/fix-leading-zeroes' 2014-08-25 13:59:19 +02:00
Philipp Hagemeister 8d31fa3cce [execafterdownload] Simplify (#3569) 2014-08-25 10:18:01 +02:00
Gjorgji Jankovski c6b4132a0a renamed for consistency 2014-08-24 18:49:04 +02:00
Gjorgji Jankovski ad260c90ab Filenames are padded according to the playlist length 2014-08-24 18:23:32 +02:00
Philipp Hagemeister be843678b1 [YouTubeDL] Correct handling of age_limit = None in result 2014-08-22 17:46:57 +02:00
Philipp Hagemeister e8ee972c6e Allow playlist test definitions in test_download.
This moves playlist tests where they belong, i.e. to the extractors themselves.
Additionally, all our network interaction configuration for tests in test_download now applies to playlist tests as well.
2014-08-21 11:52:07 +02:00